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Executive  Summary 


•  This  final  technical  report  covers  the  period  from  August  1,  1992  through 
August  31,  1995.  The  report  describes  progress  made  in  the  development  of 
the  DesignPro  interactive  computer-based  advisory  system  for  user-computer 
interface  (UCI )  design,  prototyping  and  evaluation. 

•  The  overall  process  includes  interaction  among  knowledge  templates  to 
develop  a  requirements  model  that,  in  turn,  helps  yield  displays  and  UCI 
routines  which,  in  turn,  suggest  a  prototyping  strategy  which,  in  turn, 
identifies  evaluation  tactics. 

•  The  DesignPro  system  supports  to  the  UCI  designer ;  it  does  not  call  for  the 
replacement  of  human  UCI  expertise  in  the  design  process.  The  methodology 
assumes  that  commercial-off-the-shelf  (COTS)  software  can  be  used  to  create 
(simulate)  an  integrated  environment  for  designing,  prototyping  and 
evaluating  interactive  user  computer  interaction  routines. 

•  The  project  was  anchored  in  the  systems  engineering  approach  to  interactive 
systems  design  and  development;  the  throwaway  — >  evolutionary 
prototyping  approach  to  validate  requirements  was  implemented.  An  initial 
prototype  was  released  in  January  of 1993;  another  in  April  of 1993  and 
another  in  October  1993;  refinements  were  made  to  the  prototype  in  January 
and  March  of 1994  and  then  again  in  April  1995,  with  a  final  prototype 
release  in  August  1995.  The  prototypes  were  used  to  validate  workstation 
requirements  and  to  communicate  what  the  system  does.  They  also  permitted 
the  integration  of  concepts,  tools,  &  COTS  software  programs  into  the  design. 

•  The  project  has  also  pursued  a  case  study  within  its  scope.  The  FLEX  case 
study  was  completed  during  this  reporting  period  and  presented  to  Rome 
Laboratory  personnel  in  July  1994.  FLEX  illustrated  how  the  UCI  design, 
prototyping,  and  evaluation  methodology  embedded  in  DesignPro  can  be  used 
to  design,  prototype  and  evaluate  varieties  of  command  and  control  interfaces. 

•  The  DesignPro  Advisory  System  permits  designers  of  user  computer 
interfaces  to  represent  requirements,  to  build  prototypes,  and  to  evaluate  their 
impact  -  all  via  a  “ workbench  ”  of  user  accessible  junctions. 

•  The  following  figure  presents  the  DesignPro  workbench.  Note  the  major 
functional  areas  and  the  system’s  ability  to  show  examples  of  the  features  that 
comprise  user  computer  interfaces  as  well  as  examples  of  off-the-shelf 
prototyping  tools  via  a  “browser  ”  capability. 


l 


•  The  project  synthesized  findings  from  a  variety  of  sources  and  disciplines  -  as 
suggested  by  the  following  figure: 


The  Project^  Analytical  Backdrop 


•  The  project’s  ultimate  payoff  will  depend  upon  the  nature  of  the  user-computer 
interface  design  applications  to  which  the  workbench  is  applied;  a  large 
number  of  applications  will  provide  insight  into  the  operational  capabilities  of 
the  interface  and  the  analytical  and  design  assumptions  upon  which  it  is  based. 

•  Ultimately  the  workbench  demonstrates  a  growing  trend  in  the  design  arena: 
the  embedding  of  more  and  more  design  expertise  in  the  knowledge-based 
software  systems  capable  of  —  in  most  cases  —  advising  designers  and  —  in  a 
few  cases  —  automating  design  processes. 
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1.0  Introduction 


The  last  decade  has  seen  the  proliferation  of  systems  that  are  highly  user- 
computer  interaction-intensive.  Yet  overall  performance  has  not  justified  the 
investments  we  have  made  in  tactical  "decision  aids,"  support  systems  ,and  larger 
information  systems.  We  still  hear  complaints  about  how  difficult  systems  are  to 
use,  how  they  provide  only  the  barest  analytical  support,  and  how  they  - 
ultimately  —  fail  to  satisfy  the  most  important  requirements. 

The  response  to  these  and  related  problems  has  been  incremental  and 
disembodied.  Perhaps  most  importantly,  we  have  failed  to  leverage  advanced 
information  technology  in  our  systems  design.  We  have  failed  to  use  technology 
to  help  define  requirements,  to  find  the  right  analytical  methods,  or  to  enhance 
user-computer  interfaces. 

We  have  not  invested  nearly  enough  in  low-cost  design  environments, 
environments  that  permit  the  rapid  testing  and  evaluation  of  ideas  ~  regardless  of 
how  controversial  they  might  seem.  Bureaucratic,  administrative  and  financial 
disincentives  surround  the  design  and  development  process:  it  is  easier  to  pursue 
incremental  fixes  than  to  challenge  whole  design  philosophies.  Low-cost  design, 
development  and  testing  environments  would  permit  assessments  about  what 
investments  make  sense  and  what  should  be  avoided. 

With  these  problems  in  mind,  we  developed  a  workstation-based  "environment" 
for  designing,  developing  (prototyping)  and  testing  new  interaction  concepts 
quickly  and  cost-effectively.  The  environment  relies  upon  the  design  principles 
anchored  in  the  discipline  of  systems  engineering,  commercial-off-the-shelf 
(COTS)  software,  and  lots  of  examples  of  user  computer  interaction  “features” 
and  examples  of  user  computer  interfaces. 

In  order  to  design  and  develop  the  kind  of  system  capable  of  supporting  the 
design,  prototyping  and  evaluation  of  simple  and  complex  user-computer 
interfaces  (UCIs),  it  was  necessary  to  adopt  an  overarching  design  and 
development  methodology,  a  methodology  that  could  (a)  guide  the  design  and 
development  of  DesignPro  and  (2)  be  appropriately  embedded  with  DesignPro 
itself.  As  suggested  briefly  above,  information  and  software  systems  engineering 
provided  this  disciplinary  framework. 

Figure  1  describes  the  project’s  analytical  backdrop: 
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Figure  1:  Project  Backdrop 


1.1  Information  &  Software  Systems  Engineering 

Systems  engineering  has  had  the  benefit  of  many  models,  life  cycles,  processes 
and  methods  honed  over  the  years  into  a  comprehensive  methodology  for  the 
design,  development,  testing,  and  maintenance  of  systems  of  all  kinds.  While  the 
emphasis  here  is  on  the  systems  engineering  of  software-intensive  systems,  the 
discipline  supports  systems  design  and  development  regardless  of  domain,’ 
regardless  of  organizational  context. 

Systems  engineering  traces  its  origins  to  the  work  of  Hall  (1962),  Chestnut  (1965* 
1967),  Chase  (1974),  Sage  (1977;  1983),  and  Blanchard  and  Fabrycky  (1981; 
1990).  More  recently,  Sage  (1992),  Blanchard  (1991),  Eisner  (1988),  and 
Chapman,  Bayhill  and  Wymore  (1992)  have  refined  the  methods,  tools 
techniques,  activities,  functions  and  purpose  of  systems  engineering. 


Along  the  way  an  overarching  Department  of  Defense  standard  evolved  —  MIL- 
STD  499 A  ~  which  describes  the  process  as  follows: 

“Systems  engineering  is  the . . .  logical  sequence  of  activities  and 
decisions  transforming  an  operational  need  into  a  description  of 
system  performance  parameters  and  a  preferred  system  configuration 
...  systems  engineering  is  the  application  of  scientific  and 
engineering  efforts  to  (a)  transform  operational  need  into  a 
description  of  system  performance  parameters  and  a  system 
configuration  through  the  use  of  an  iterating  process  of  definition , 
synthesis,  analysis,  design,  test,  and  evaluation;  (b)  integrate  related 
technical  parameters  and  ensure  compatibility  of  all  physical, 
functional,  and  program  interfaces  in  a  manner  that  optimizes  the 
total  system  definition  and  design;  (c)  integrate  reliability, 
maintainability,  safety,  survivability,  human,  and  other  such  factors 
into  the  total  engineering  effort  to  meet  cost,  schedule,  and  technical 
performance  objectives.  ” 

The  revised  standard  (DOD,  1992)  describes  the  systems  engineering  process  as 
follows: 

"A  comprehensive,  iterative  problem  solving  process  that  is  used  to: 

(a)  transform  validated  customer  needs  and  requirements  into  a  life- 
cycle  balanced  solution  set  of  system  product  and  process  designs,  (b) 
generate  information  for  decision-makers,  and  (c)  provide 
information  for  the  next  acquisition  phase. " 

Sage  (1992)  offers  three  definitions  of  systems  engineering  which  constitute 
perspectives  on  the  process.  He  defines  systems  engineering  structurally, 
functionally,  and  in  terms  of  purpose.  Structurally,  Sage  see  systems  engineering 
as  a  "management  technology  to  assist  clients  through  the  formulation,  analysis, 
and  interpretation  of  the  impacts  of  proposed  policies,  controls  or  complete 
systems  upon  the  perceived  needs,  values,  institutional  transactions  of 

stakeholders." 

Functionally,  systems  engineering  -  according  to  Sage  (1992)  --  is  an 
"appropriate  combination  of  theories  and  tools,  carried  out  through  the  use  ot  a 
suitable  methodology  and  set  of  systems  management  procedures."  ^ 

Sage's  purposeful  definition  of  systems  engineering  assumes  that  the  purpose  of 
systems  engineering  is  information  and  knowledge  organization  that  will  assist 
clients  who  desire  to  develop  policies  for  management,  direction,  control  and 
regulation  activities  relative  to  forecasting  planning,  development,  production  and 
operation  of  total  systems  to  maintain  overall  integrity  and  integration  as  related 
to  performance  and  reliability." 
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Eisner  (1988)  defines  systems  engineering  as  “an  iterative  process  of  top-down 
synthesis,  development,  and  operation  of  a  real-world  system  that  satisfies,  in  a 
near  optimal  manner,  the  full  range  of  requirements  for  the  system.”  Eisner 
(1988)  describes  the  systems  engineering  process  as  consisting  of  the  following 
“elements:” 

1.  Requirements  analysis; 

2.  Requirements  allocation; 

3.  Functional  analysis; 

4.  Functional  allocation ; 

5.  Specification  analysis; 

6.  Specification  development; 

7.  Preliminary  design: 

8.  Interface  definition; 

9.  Schedule  development; 

10.  Preliminary  cost-analysis; 

11.  Technical  performance  measurement; 

12.  Trade -off, '/alternative  analysis; 

13.  Pre-planned  product  improvement; 

14.  Final  design: 

15.  Schedule  update; 

16.  Cost  update; 

17.  Fabrication; 

18.  Coding; 

19.  Preliminary  testing; 

20.  Debugging  &  reconfiguration; 

21.  Testing  &  integration; 

22.  Updates: 

A.  Schedule; 

B.  Cost; 

C.  Technical  performance  measurement; 

23.  Documentation; 

24.  Training;  and 

25.  Production. 

Eisner’s  generic  systems  engineering  process  appears  in  Figure  2.  Note  the 
emphasis  on  “front-end”  activities,  activities  that  determine  development  and 
maintenance  requirements.  The  importance  that  systems  engineers  place  on  this 
front-end  process  is  precisely  the  emphasis  that  defines  the  purpose,  embedded 
processes  and  outcome  of  our  knowledge-based  advisory  workbench  — 
DesignPro. 
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Figure  2:  Eisner’s  Generic  Systems  Engineering  Process 


Blanchard’s  (1991)  systems  engineering  process  combines  these  and  other  phases, 
steps  and  elements  into  a  process  that  we  can  use  to  link  to  various  alternative 
systems  engineering  process  models  and  life  cycles. 


1.1.1  Systems  Engineering  Goals 

The  software  systems  engineering  process  is  implemented  in  order  to  achieve 
certain  objectives.  Sage  (1992)  lists  the  following: 

•  All  (life  cycle )  encompassing 

•  Problem  understanding 

•  Communication 

•  Early  capture  of  design  &  implementation  needs 

•  Bottom-up  &  top-down  design  &  development 

•  Alternative  systems  management  approaches 

•  Process  &  product  quality  assurance 

•  Product  evolution 

•  Support  for  configuration  management  standards 

•  Support  for  automated  design  &  development  aids 

•  Teachable  &  transferable  methodology 

•  All  phase  definition  &  documentation 

•  Operational  product  functionality,  revisability  &  transitioning 

•  Support  system  product  development  &  system  user  organizations 
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The  thrust  of  this  list  is  that  successful  systems  engineering  should  be  disciplined, 
structured,  informed  by  knowledge  and  data,  repeatable,  documented,  and  well 
managed.  In  effect,  the  argument  here  is  for  success  through  process  maturity. 
But  perhaps  more  importantly,  Sage's  definitions  of  systems  engineering  speak 
directly  to  objectives  and  goals.  In  other  words,  the  objectives  of  the  systems 
engineering  process  include  attention  to  the  structure ,  function ,  and  purpose  of 
systems  engineering. 


1.1.2  System  Definition 

The  essence  of  the  systems  engineering  process  is  requirements  — >  design  — > 
development  efficiency.  The  primacy  of  requirements  is  well  documented 
(Andriole,  1989,  1990;  Sage,  1992;  Sage  &  Palmer,  1990;  Davis,  1989,  1992). 
Eisner’s  process  model  serves  us  well  again  here.  It  focuses  directly  on  the 
"definition  of  need"  and  the  development  of  a  preliminary  design  via  require¬ 
ments  analysis,  modeling,  and  allocation,  trade-off  analysis,  and  input  to  the 
detailed  design  and  development  process. 

Mainstay  requirements  tools  include  functional  block  and  flow  diagrams  which 
identify,  define  and  communicate  the  functions,  tasks  and  sub-tasks  that  the 
system-to-be  should  perform.  The  specification  of  a  system's  external  behavior  is 
endemic  to  the  overall  systems  engineering  process;  in  fact,  external  behavior 
specification  is  a  filter  through  which  subsequent  design  and  development 
decisions  should  be  passed.  From  another  perspective,  external  behavior 
specification  is  a  gate  that  cannot  be  traversed  until  consensus  about 
functional  capabilities  emerges.  Other  tools  include  NxN  charting,  hierarchical 
task  models,  and  data/control  flow  models  (Sailor,  1990;  Eisner,  1988). 

In  addition  to  these  tools  are  a  variety  of  others.  Eisner  (1989)  describes 
dependency  diagrams,  signal-flow  block  diagrams,  Petri  Nets,  hierarchical  input- 
process-output  diagrams,  Wamier-Orr  diagrams,  Michael  Jackson  diagrams, 
action  diagrams,  sequence  and  timing  diagrams,  parameter  dependency  diagrams, 
logic  flow  charts,  Nassi-Shneiderman  charts,  and  decision-network  diagrams, 
among  others.  These  and  related  methods,  tools  and  techniques  —  many  of 
which  are  computer-based  -  permit  systems  engineers  to  define,  model,  and 
validate  requirements  (prior  to  expensive  design  or  development  activities). 

It  is  essential  to  remember  that  systems  engineers  expect  conflict  during  the 
requirements  process.  They  expect  requirements  to  evolve  over  time  as  they  are 
discovered  by  the  systems  analysis  team.  But  they  also  expect  a  prioritized 
requirements  hierarchy  to  emerge  early  on  in  the  process. 
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Risk  also  becomes  a  major  issue  during  the  front-end  requirements  analysis, 
modeling  and  prototyping  process.  It  is,  in  practice,  another  side  of  trade-off 
analysis,  where  stakeholders  are  asked  to  prioritize  and  then  assess  feasibility.  If 
a  high  important  requirement  cannot  easily  be  satisfied  then  risk  rises 
dramatically.  Risk  also  rises  when  technological  feasibility  is  assessed  "low."  It  is 
not  unusual,  for  example,  for  system  plans  to  call  for  the  insertion  of  a  specific 
technological  capability  at  some  point  in  the  systems  life  cycle.  But  what  if  the 
capability  does  not  exist?  How  serious  might  delays  become?  Should  investments 
be  made  today  to  engineer  a  capability  tomorrow?  Risk  assessment  is  essential  to 
system  success,  since  early  miscalculations  almost  always  cost  a  great  deal  of  time 
and  money  downstream. 

The  key  to  the  process  is  early  diagnosis  of  key  requirements,  feasibility,  and 
risk.  When  the  results  of  these  processes  suggest  conflict  then  trade-off  analysis 
ensues.  Here  the  systems  engineer  calculates  -  via  empirical  and  subjective  data 
-  the  relationships  among  competing  requirements  and  the  cost-benefit  calculus 
derived  from  the  relationships.  Where  requirements  prioritization  can  yield  a 
"benefit-only"  list  of  requirements,  trade-off  analysis  should  yield  a  cost-benefit- 
based  list  (where  cost  is  defined  as  time,  money  and  talent). 

Good  systems  engineering  requirements  that  system  requirements  be  identified, 
modeled,  validated,  discovered,  and  "traced"  throughout  a  system's  life.  Data 
regarding  the  source  of  requirements,  their  original  form,  their  relative 
importance,  their  flexibility,  and  their  precise  location  in  prototypes  all  must  be 
stored  for  easy  access  to  the  systems  engineering  team. 

All  of  this  front-end  work  is  intended  to  deepen  our  understanding  of  initial 
requirements,  to  prevent  major  requirements  from  falling  through  project 
cracks,  and  to  provide  insight  into  the  next  -  prototyping  -  phase  of  the  front- 
end  systems  engineering  process. 

Requirements  are  thus  validated  with  reference  to  constraints  and  alternative 
designs.  Trade-off  analysis,  risk  analysis  and  other  forms  of  "sensitivity  analysis" 
are  conducted  to  determine  how  diagnostic  requirements  are,  can  realistically  be, 
and  might  be  given  additional  investments. 

The  functional  outcome  of  this  process  is  a  system  synthesis  and  definition  of 
sufficient  diagnosticity  to  permit  more  detailed  design  and  development. 


1.1.3  Design  &  Development 

The  detailed  design  and  development  process  (a)  assumes  that  the  system 
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definition  that  emerges  from  the  front-end  process  is  sound  and  (b)  that  the  steps 
toward  production  are  clear.  System/product  design,  prototype  development,  and 
prototype  testing  &  evaluation  constitute  the  detailed  design  and  development 
process. 

The  design  and  development  process  can  be  accelerated  via  prototyping.  But 
prototyping  is  most  effective  when  requirements  are  reasonably  well  understood, 
system  trades  have  been  performed,  and  a  system  concept  likely  (but  obviously 
not  certain)  to  satisfy  requirements  has  emerged.  There  are  several  forms  worth 
noting:  throwaway,  evolutionary  and  hybrid  prototyping.  Throwaways  are  used 
for  new  system  concepts,  while  evolutionary  prototypes  are  used  when 
requirements  are  much  better  understood  or  when  a  well-documented  system  is 
under-going  enhancement.  Hybrids  are  used  when  parts  are  well  understood  and 
some  parts  are  not. 

Systems  engineers  prototype  routinely;  they  also  evaluate  the  prototypes  to 
determine  the  extent  to  which  they  satisfy  functional  requirements  as  well  as 
specific  non-functional  requirements,  such  as  security,  modifiability,  and 
transportability.  Dorfman  (1990),  among  many  others,  suggests  that  prototyping 
should  precede  systems  and  software  design,  that  the  performance  and  evaluation 
of  the  prototype  should  inform  the  design  and  development  process.  Prototypes 
can  —  and  should  —  be  evaluated  with  reference  to  their  usability,  the  degree  to 
which  they  satisfy  functional  and  non-functional  requirements,  and  likely 
implementation  costs. 


1.1.4  Systems  Engineering  Management 

The  systems  engineering  process  must  be  carefully  managed.  By  definition  the 
process  is  complex  and  management  quickly  becomes  essential  as  the  phases  of  the 
life  cycle  are  implemented. 

Eisner  (1989)  assumes  that  at  the  highest  level  the  systems  engineering  process 
itself  must  be  managed,  that  engineering  speciality  areas  must  be  identified  and 
managed,  and  that  technical  program  planning  and  control  is  essential.  Sage 
(1992)  expands  the  concept  to  include  organizational  management  and  structures, 
quality  assurance  via  configuration  management,  reviews  and  standards,  and 
strategic  quality  assurance  and  management. 

Blanchard  (1991)  suggests  that  systems  engineering  management  can  be  defined 
around  the  systems  acquisition  process  and  the  major  milestones  that  track  with 
the  major  phases  of  the  systems  engineering  process.  Blanchard's  management 
model  identifies  the  need  for  a  formal  systems  engineering  management  plan 
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(SEMP),  a  test  and  evaluation  (T&E)  master  plan  (TEMP),  and  various  design 
reviews.  These  plans  and  reviews  are  incarnated  in  "specifications,"  such  as  the 
venerable  “  ’A’  spec.” 

In  addition,  systems  engineering  management  involves  decision-making  that 
occurs  as  a  result  of  feedback  from  specific  kinds  of  analyses,  such  as  risk 
analysis,  technology  assessment,  and  the  like.  Systems  engineering  management 
ideally  locates  "off-ramps"  in  the  design  and  development  process,  off-ramps  that 
inform  —  and  sometimes  delay  or  cease  --  the  design,  development  and 
production  process.  The  ideal  management  process  is  one  that  is  analytical, 
adaptive  and  decisive. 

Sage  (1992)  suggests  that  systems  engineering  management  can  be  defined  around 
audits,  reviews,  standards,  and  systems  integration  --  all  with  reference  to  quality. 

All  of  these  management  concepts  and  objectives  travel  with  the  systems 
engineering  process  itself.  Structurally,  systems  engineering  is  --  according  to 
Sage  —  technology  management.  The  Department  of  Defense  Standards  all  find  it 
difficult  to  separate  management  from  technology  and  engineering. 


1.1.5  Systems  Engineering  &  DesignPro 

It  is  important  to  note,  however,  that  our  view  of  systems  engineering  is  not 
without  application  focus.  We  are  concerned  primarily  with  the  systems 
engineering  of  software-intensive  systems.  We  are  also  concerned  with  the 
systems  engineering  of  individual  software-intensive  products  and  product  lines, 
and  as  an  overarching  organizational  planning  and  management  process. 

The  software  systems  engineering  life  cycle  adopted  here  appears  below.  It  was 
adapted  from  the  generic  systems  engineering  process: 

1.  Analyze  Requirements 

A.  Method(s):  Interview,  Observe,  Simulate  ... 

B.  Tool(s):  Off-The-Shelf  Software:  Outline/Idea  Processors  ... 

C.  Objective:  First  Cut  at  User  Requirements;  Document  for 
Continued  Use  ... 

2.  Model  Requirements 

A.  Method(s):  Hierarchical  Decomposition,  IPO,  Causal, 
Cognitive  Maps/Mental  Models,  Simulations  ... 


13 


B.  Tool(s):  OTS  Software:  Hierarchical  Tools,  Simulation  Tools, 
General  Purpose  Modeling  Tools  ... 

C.  Objective:  Initial  Transition  from  Observational  &  Textual  Data 
&  Information  into  Structured,  Organized  Model  of  User 
Requirements  ... 

3.  Assess  Constraints 

A.  Methods(s):  Cost-Benefit  Analysis,  Impact  Assessments, 

Multi- Attribute  Utility  Assessment  (MAUA),  Qualitative  & 
Quantitative  Modeling  ... 

B.  Tool(s):  OTS  Software:  Decision  Analytic  Packages,  MAUA 
Tools,  Modeling  Packages ... 

C.  Objective:  Identify  Key  Obstacles  to  Development  & 
Implementation;  Measure  the  Impact  with  Reference  to 
Requirements  ... 

4.  Prioritize  Requirements 

A.  Methods(s):  Qualitative  &  Quantitative  Rank-Ordering; 
Simulation-Based  Prioritization  ... 

B.  Tool(s):  OTS  Software:  Hierarchical  Decomposition  Packages; 
MAUA  Packages ... 

C.  Objective:  Initial  List  of  Most  Important  Requirements 
Given  Constraints  &  Initial  Requirements  Model ... 

5.  Develop  Alternative  System  Concepts 

A.  Methods(s):  Cost-Benefit  Analysis,  Impact  Assessments, 
Requirements  Tracing  ... 

B.  Tool(s):  OTS  Software:  Decision  Analytic  Packages, 

Simple  Configuration  Models  ... 

C.  Objective:  Identify  System  Concepts  Most  Likely  to  Satisfy 
Prioritized  Requirements  ... 

6.  Trade-Off  Analysis 

A.  Methods(s):  Cost-Benefit  Analysis,  Impact  Assessments, 

Multi- Attribute  Utility  Assessment  (MAUA) ... 

B.  Tool(s):  OTS  Software:  Decision  Analytic  Packages, 

MAUA  Tools... 

C.  Objective:  Identify  Key  Obstacles  to  Development  & 
Implementation;  Measure  the  Impact  with  Reference  to 
Requirements  ... 
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7.  System  Concept  Specification 

A.  Methods(s):  Cost-Benefit  Analysis,  Impact  Assessments, 
Requirements  Tracing,  Sizing  ... 

B.  Tool(s):  OTS  Software:  Decision  Analytic  Packages, 

Simple  Configuration  Models,  Process  Models  ... 

C.  Objective:  Identify  System  Concepts  Most  Likely 
to  Satisfy  Prioritized  Requirements  ... 

8.  Prototype  Requirements 

A.  Method(s):  Throwaway/Evolutionary:  Screens,  Interactions, 
Output  (External  Behavior  of  System)  ... 

B.  Tool(s):  OTS  Software:  Screen  Formatters,  Interactive 
Prototypers,  Code  Generators,  UIMSs  ... 

C.  Objective:  Throwaway  Prototype:  Convert  Requirements  Model 
into  a  System  Concept  that  Demonstrates  UCI  and  Overall 
Functionality;  Evolutionary  Prototype:  Begin  Transition  to  Full 
Scale  Production  (Build  a  Little,  Test  a  Little)  ... 

9.  Specify  Software  Requirements 

A.  Method(s):  DFDs,  ERDs,  NSs,  GS,  DeMarcos  ... 

B.  Tool(s):  OTS  Software:  Conventional  CASE  Tools  ... 

C.  Objective:  Represent  the  Software  Design  Via  Consistent 
Notation  to  "Guide"  Production  of  Code  &  Document  the 
Conversion  Process  ... 

10.  Design  &  Develop  Software 

A.  Method(s):  Structured  Programming,  Programming 
"Conventions"  ... 

B.  Tool(s):  Programmer  Workbenches;  CASE  Tools; 

Programming  "Environments"  (e.g.,  Ada)  ... 

C.  Objective:  Software  that  Works  ... 

11.  Test  Software 

A.  Method(s):  Qualitative  &  Quantitative  Methods  ... 

B.  Tool(s):  CASE  Tools;  Unit  Testing  Tools;  Diagnostic  Tools  ... 

C.  Objective:  Verified  &  Validated  Software  ... 

12.  Field  System  ... 
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1.2  The  Primacy  of  Requirements,  Prototyping  &  Evaluation 

The  whole  project  was  anchored  in  systems  engineering  which  itself  is  anchored 
in  the  primacy  of  requirements.  Good  systems  engineers  place  requirements  at 
front  and  center  of  the  design  process.  Our  project  did  precisely  the  same  thing: 
we  assumed  that  the  best  user  computer  interface  would  be  one  informed  by  user 
requirements,  requirements  that  could  be  validated  via  prototyping.  In  order  to 
determine  the  likely  impact  of  specific  UCI  features,  the  project  (and  its 
DesignPro  workbench)  also  supports  evaluation,  which  is  important  is  our  on¬ 
going  effort  to  prevent  the  programming  of  interfaces  unlikely  to  enhance 
human-computer  interaction. 


1.3  User-Computer  Interface  Design,  Prototyping  &  Evaluation 

The  project  assumed  that  it  was  possible  to  identify  the  steps  that  together 
comprise  the  UCI  design,  prototyping  and  evaluation  process.  We  further 
assumed  that  it  was  possible  to  model  this  process  via  the  synthesis  of  the  systems 
engineering  process  and  knowledge  bases  placed  at  various  locations  in  the  design 
process.  The  simple  concept  appears  in  Figure  3: 
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Figure  3:  Simple  Interaction  Process 


A  more  detailed  view  appears  in  Figure  4.  This  view  indicates  how  requirements 
can  be  modeled  and  then  “converted”  via  interaction  with  a  knowledge  base 
(comprised  of  simple  rules  about  the  relationships  among  task  requirements  and 
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UCI  features)  into  UCI  feature  recommendations  which,  in  turn,  can  be 
prototyped  for  subsequent  evaluation. 

The  evaluation  process  sheds  light  onto  the  prototyping  process.  If  the  prototype 
inaccurately  reflects  user  requirements  then  the  evaluation  will  yield  inconclusive 
or  negative  results;  on  the  other  hand,  if  the  prototype  enhances  human  computer 
performance,  then  the  requirements  have  probably  been  faithfully  represented 
throughout  the  design  process. 

Figure  4  illustrates  graphically  what  the  process  looks  like: 


Figure  4:  The  Knowledge-Based  Requirements  — >  Prototyping  — >  Evaluation  Process 
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1.4  Project  Overview 


The  project  to  incarnate  UCI  design,  prototyping  and  evaluation  knowledge  and 
experience  (via  examples  and  COTS  tools),  began  in  1992.  The  project  was 
completed  in  the  summer  of  1995.  Project  tasks  are  identified  below  in  Figure  5. 
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Figure  5:  Project  Tasks 
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1.5  Project  Results 

A  prototype  --  known  as  DesignPro  --  was  developed,  a  major  case  study  was 
undertaken,  a  new  methodology  was  refined,  and  the  project  was  documented. 


1.5.1  Knowledge-Based  UCI  Design,  Prototyping  &  Evaluation 
The  objective  of  the  project  was  the  representation  of  the  systems  engineering 
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Figure  6:  The  Design  &  Prototyping  Process 
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design  process  in  an  application  that  would  support  the  systems  engineering  of 
user-computer  interfaces  and  interaction  routines  --  as  suggested  above  by  Figure 
6. 


The  major  design  process  (and  workbench)  output  includes  user-computer 
interface  designs,  interactive  prototypes,  and  prototype  evaluations;  these 
“products”  are  developed  with  the  assistance  of  a  knowledge  base  that  can 
accelerate  the  design  — >  prototyping  — >  evaluation  process. 


1.5.2  The  FLEX  Case  Study 

The  development  of  the  Force-Level  Execution  (FLEX)  prototype  at  the  Air 
Force  s  Rome  Laboratory  (RL)  presented  an  excellent  opportunity  for  applying 
and  evaluating  the  our  methodology  for  UCI  design.  The  FLEX  Program  was  a 
collaborative  rapid  prototyping  effort  between  the  Advanced  Concepts  Branch  in 
the  RL/C3  Division  and  industry  contractors.  Officers  from  the  US  Air  Force’s 
major  operational  commands  in  the  United  States,  Europe,  Pacific  and  the  Far 
East  participated  in  a  review  board  known  as  the  FLEX  Working  Group  (FWG) 
to  provide  an  operational  end-user  perspective.  The  RL  development  team  took 
responsibility  for  developing  the  user  interface. 

The  Drexel  University/George  Mason  University  team  worked  within  this  design 
and  prototyping  organization  to  enhance  the  user-computer  interface  and 
interaction  process  design  and  prototyping  process.  This  work  resulted  in  a 
prototype  that  enhanced  performance  substantially,  and  suggested  the  direction  in 
which  implementation  should  proceed  (precisely  the  goal  of  the  design 
methodology  and  the  DesignPro  system  which  incarnates  the  methodology). 

The  design  methodology  was  employed  to  define  the  problem,  identify  and  model 
the  (cognitive)  task  requirements,  integrate  the  requirements  into  the 
System/Segment  Specification  (SSS),  translate  requirements  into  UCI  design 
goals,  create  an  interactive  prototype  of  the  FLEX  interface,  and  develop  a  plan 
to  evaluate  the  prototype  against  the  existing  FLEX  interface. 

Section  3.0  of  this  report  provides  detailed  information  about  the  FLEX  case 
study. 


1.5.3  The  ( Cognitive )  Task  Analysis  Process 

A  major  outcome  of  the  project  was  the  development  of  an  entire  methodology 
for  identifying  and  modeling  tasks,  tasks  that  feed  the  larger  user  and  system 
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requirements  modeling  process.  The  step-by-step  methodology  emerged  from 
the  project,  a  methodology  that  was  developed  into  a  Handbook  for  conducting 
tasks  analysis-based  requirements  analyses  and  modeling,  prototyping  and 
evaluation. 

This  Handbook  appears  in  Appendix  C. 


1.5.4  The  Knowledge-Based  UCI  Design ,  Prototyping  &  Evaluation 
Workbench 

The  knowledge-based  system  -  DesignPro  --  that  resulted  from  the  research  and 
development  currently  runs  on  Apple  Power  Macintoshes  (and  other  high  end 
Macintoshes).  The  system  supports  user  computer  interaction  designers  as  they 
identify  user  requirements  (defined  as  “tasks”)?  build  interactive  prototypes  (via 
the  implementation  of  embedded  COTS  software),  and  evaluate  the  prototype(s) 
to  determine  if  their  features  should  be  implemented  in  production  code. 

Section  4.0  of  this  report  describes  DesignPro  workbench  in  detail,  while 
Appendix  A  presents  the  screens  from  the  system. 
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2.0  UCI  Design ,  Prototyping  &  Evaluation 


The  project  assumes  --  as  discussed  above  ~  that  the  best  insight  into 
requirements  that  should  be  satisfied  in  formal  designs  and  development  efforts 
can  be  obtained  from  a  structured,  repeatable  requirements  analysis,  prototyping, 
and  evaluation  process. 


2.1  UCI  Requirements  Analysis 

The  project  assumed,  therefore,  that  effort  should  be  directed  to  the  front  part  of 
the  front-end  of  the  design  and  development  process:  user  requirements.  The 
form  of  those  requirements  is  the  task.  Task  analysis  is  well  established  as  a 
requirements  notation.  It  is  also  consistent  with  our  goals  to  repeat,  model, 
trade-off  and  prioritize  requirements. 

Task-based  requirements  modeling  permits  reusability  and  is  consistent  with 
current  efforts  to  use  objects  to  support  design  and  development. 

Task-based  requirements  also  permits  requirements  documentation  via 
descriptions  of  user  tasks  that  can  be  assessed  and  prioritized. 

While  there  are  certainly  alternatives  to  task-based  requirements  modeling,  we 
opted  for  this  flexible  approach  because  the  alternatives  did  not  satisfy  the  design 
requirements  unique  to  user-computer  interface  and  interaction  routine  design 
and  development.  UCI  design  is  inherently  task-oriented.  This  is  because  of  the 
linkage  to  UCI  display  and  interaction  features  and  how  they  can  be  traced  to 
specific  and  groups  of  user  tasks  that  the  system  needs  to  support  (to,  in  turn, 
support  the  functions  that  the  define  the  system  and  the  purpose  for  which  the 
system  exists). 

Our  use  of  tasks  is  intended  to  “trickle  up”  to  functionality.  Our  requirements 
template  assumes  that  functional  requirements  (along  with  non-functional  and 
purposeful  requirements)  will  define  the  specification  and  design  process.  Tasks 
lead  to  functions  which  lead  to  purpose.  This  is  the  track  that  DesignPro  follows. 

The  essence  of  the  design  process  --  and  therefore  what  DesignPro  supports  -  is 
requirements  modeling.  Specific  effort  was  taken  to  ensure  that  the  “front-end  of 
the  front-end”  was  solid.  While  we  have  obviously  made  some  analytical  choices, 
we  believe  that  the  existing  task-based  requirements  modeling  process  will  yield  * 
diagnostic  requirements  data,  data  that  can  be  fed  directly  into  the  design  and 
prototyping  processes. 
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Figure  7  presents  the  elements  that  together  yield  a  requirements  model  (which, 
in  turn,  becomes  the  input  to  the  design  and  prototyping  processes). 


Domain  Characteristics 


Requirements  Model 


Figure  7:  The  Requirements  Modeling  Process 
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2.2  UCI  Features 

The  DesignPro  system  provides  support  for  the  identification  of  the  UCI  design 


Displays  &  UCI  Routines 


Figure  8:  The  Display  &  UCI  Routine  Identification  Process 
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features  that  are  most  likely  to  satisfy  user  requirements  (given  the  constraints 
identified  during  the  requirements  modeling  phase). 

As  Figure  8  suggests,  there  are  a  number  of  elements  that  lead  to  the 
recommended  displays  &  UCI  routines. 


2.3  UCI  Prototyping 
DesignPro  supports  prototyping. 

Prototyping  is  an  absolute  prerequisite  to  writing  software  requirements 
specifications.  You  may  think  you  know  the  requirements  (even  for  a  system 
enhancement),  but  the  very  best  you  can  expect  to  do  is  build  and  test 
evolutionary  prototypes. 

The  process  we  advocate  (and  have  embedded  in  DesignPro)  -  and  the  one  that 
virtually  always  applies  --  is:  prioritized  requirements  (given  constraints)  — > 
throwaway  prototyping  — >  initial  software  requirements  —  >  evolutionary 
prototyping  —  >  detailed  software  requirements  specifications. 

Detailed  software  requirements  specifications  should  not  —  in  our  experience  -- 
emerge  without  prototyping  feedback.  This  is  obviously  true  when  throwaway 
prototypes  are  built,  but  also  necessary  during  the  initial  evolutionary  iterations. 
One  can  never  assume  that  requirements  data  is  valid;  it  must  be  inspected  —  and 
demonstrated  -  by  lots  of  eyes,  lots  of  perspectives,  and  lots  of  objectivity. 

The  key  lies  in  "templating"  the  requirements  modeling  and  prototyping  process 
and  in  getting  experienced  professionals  to  implement  and  manage  the  process. 
The  key  also  lies  in  self-documenting  COTS  software  that  permits  group  design 
and  communication. 

Another  key  is  pragmatism.  It's  essential  that  we  appreciate  the  fluid,  changing 
nature  of  requirements,  and  the  role  that  prototyping  plays  in  the  requirements 
discovery  process. 

DesignPro  supports  the  iterative  prototyping  process  via  the  application  of  COTS 
software  (which  is  embedded  in  the  system). 

The  architecture  of  the  system  permits  the  addition  (or  deletion)  of  COTS 
software  as  it  becomes  available  or  as  new  tools  are  identified  as  “preferred.” 
However,  it  must  be  remembered  that  not  all  COTS  tools  are  equal,  and  that  some 
are  better  suited  to  do  some  features  prototyping  and  some  less  so. 
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Figure  9  presents  the  prototyping  “template.” 


Interactive  Prototypes 


Figure  9:  The  Prototyping  Process 
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2.4  UC I  Evaluation 

As  Figure  10  suggests,  the  evaluation  process  is  also  straightforward. 


Figure  10:  The  Evaluation  Process 
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2.5  Knowledge-Based  UCI  Design ,  Prototyping  &  Evaluation 

The  Imowledge  base  lies  at  the  heart  of  the  system.  The  approach  taken  to  the 
identification  of  design  processes  is  anchored  in  the  generic  “objects-attributes- 
values”  approach  to  knowledge  representation. 

Figure  11  presents  the  approach  graphically. 


Task  User 

Characteristics  T  Characteristics 


Constraints 
&  Context 


Design 

Recommendat  ions 
(Via  "Templates," 
Via  Video  Snippets, 
Via  Textual 
Explanations  & 
Via  Case  Studies ... ) 


Prototyping  -J-  Evaluation 
Recommendations  Recommendations 

(Via  "  T emplates, "  (Via  Methods 

Via  Access  to  Recommendations, 

COTS  Software  Via  Experimental 

&  Recommendations,  Designs  &  Via 

Via  Textual  Case  Studies ... ) 

Explanations  & 

Via  Case  Studies ... ) 


Figure  11 :  The  Knowledge  Base  Structure 

The  knowledge  base  itself  was  developed  from  several  sources:  the  case  study 
literature,  the  experience  of  the  design  team,  and  the  unique  aspects  of  the  domain 
that  “interpret”  the  relationships  among  the  objects,  attributes  and  values. 

The  design  and  prototyping  recommendations  DesignPro  generates  can  be  traced 
to  these  relationships  and  to  the  characteristics  of  the  domain  that  we  focused 
upon:  command  &  control. 
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3.0  The  FLEX  Case:  Feature  Enhancements  Via  Prototyping 


3.1  Defining  the  Problem 

The  first  step  in  problem  involved  defining  the  FLEX  system  and  placing  it  in 
context  within  the  organization.  Most  of  the  initial  definition  was  based  upon  an 

early  version  of  the  FLEX  System/Segment  Specification  (SSS)^  and  the  trip 
reports  written  by  the  RL  development  team  after  their  visits  to  air  operations 
centers  in  the  United  States,  Europe,  the  Pacific  and  the  Far  East.  Additional 
information  was  drawn  from  the  demonstration  of  the  first  FLEX  prototype  for 
the  FWG  members. 

As  indicated  in  Figure  12,  FLEX  is  part  of  a  suite  of  systems  that  supports  the 
Combat  Operations  Division  (COD)  of  the  Air  Operations  Center  (AOC)  in  the 
planning  and  execution  of  the  air  missions.  FLEX  receives  the  mission  plan  from 
the  Combat  Plans  Division  in  the  form  of  an  Air  Tasking  Order  (ATO). 

Formatted  in  machine-readable  text,  an  ATO  for  24  hours  of  combat  missions 
may  run  into  hundreds  of  pages.  While  the  text  form  permits  rapid  transmission 
to  the  operational  units,  the  ATO  is  unwieldy  and  does  not  provide  a  sense  of  the 
actual  mission  flows  (Figure  13).  To  compensate,  planners  and  operational  staff 
officers  manually  create  a  variety  of  charts  and  maps  to  display  mission  data  in  a 
form  that  permits  multiple  data  views.  The  Advanced  Planning  System  (APS) 
allows  the  planning  team  to  work  with  the  details  of  the  ATO  data  using  tabular 
displays  of  the  relevant  data  bases  and  automating  many  of  the  calculation  and 
charting  operations.  Since  the  planners  and  operational  decision-makers  use 
much  of  the  same  data  and  knowledge  bases,  APS  and  FLEX  share  many  common 

screen  layouts.  ^  The  common  windows  help  promote  consistency  across  these 
two  closely-coupled  systems. 

The  combat  air  operations  decision  environment  is  complex  and  dynamic, 
involving  a  high  degree  of  uncertainty  combined  with  time  pressure  and  high 
threat.  The  duty  officers  (DOs)  in  the  COD  monitor  the  execution  of  the  ATO 
missions  and  re-plan  as  required  to  meet  changes  in  goals  and/or  available 

resources. ^  The  various  air  missions  are  so  interdependent  that  changes  in  the 
availability  of  a  support  mission  can  result  in  the  cancellation  or  re-scheduling  of 
attack  and  support  missions  across  the  entire  ATO.  This  “ripple  effect”  makes 
timely  re-planning  extremely  difficult. 

1  The  SSS  is  the  standard  document  for  system-level  requirements  documentation  under  the  Dept, 
of  Defense  software  development  standard,  DOD-STD-2167a. 

2  The  Duty  Officer  in  AOC  is  a  decision-maker,  thus,  in  discussions  of  FLEX,  the  term  decision¬ 
maker  (DM)  is  used  interchangeably  with  duty  officer  (DO). 
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Figure  12:  FLEX  &  the  ATO  Timeline 
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Overlapping  ATOs 
in  Combat  Ops 
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/12 
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/JP4/MID 

LATE 
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/NOSEGAY 

11/2A10A 

112 
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/JP4/PRE 

/62 

LATE 
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11/2A10A 

/12 
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/ 

/JP4/MID 

// 

// 

TASKUNIT/353TFS/OEPA// 

MSNDAT/46/-/NOSEBAG01/1A10A/XCAS/-/B5/-/124/23301/33301// 
MSNLOC/150800Z/151000Z/HANDEL/ALT:200/-/2840N05535E// 
REFUEL/ROM  AN00/2 1  /ESSO/ALT:  1 70/-/1 0// 

REFUEL/ROM  AN00/2 1/ESSO/ ALT:  1 70/-/1 1  // 

REFUEL/ROM  AN00/21 /ESSO/ALT:  170/  1211 


Figure  13:  Examples  from  an  Unclassified  AID 


The  first  models  developed  for  problem  definition  decomposed  the  monitoring 
and  control,  planning,  and  communications  tasks  performed  by  the  COD 
decision-makers  in  terms  of  their  representation  in  the  first  FLEX  prototype  and 
the  trip  reports.  These  models  were  iteratively  refined  as  the  requirements  were 
identified. 

To  provide  a  tractable  example,  the  case  study  focused  only  on  the  FLEX  re¬ 
planning  support  to  the  Tanker  Duty  Officer  (TDO).  The  TDO  is  responsible  for 
providing  air  refueling  (AR)  support  to  all  scheduled  missions  that  require 
refueling.  Re-planning  is  required  when  new  missions  are  created,  existing 
missions  re-routed,  or  air  refueling  resources  change.  The  TDO  performs  re¬ 
planning  tasks  as  indicated  by  their  own  assessment  of  the  evolving  situation  and 
as  tasked  by  other  duty  officers.  Although  these  models  were  roughed  out  during 
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the  problem  definition  phase,  most  of  the  detail  was  developed  as  part  of  the  in- 
depth  analysis  conducted  during  the  requirements  identification  phase. 

3.2  Identifying  and  Modeling  the  Tanker  Duty  Officer's  Cognitive 

Task  Requirements 

The  case  study  was  external  to  the  actual  FLEX  development  effort;  therefore, 
the  task  requirements  identification  process  began  with  the  examination  of  system 
requirements  information  gathered  from  a  variety  of  sources  including: 

•  Document  Reviews  -  Rome  Laboratory  (RL)  development  team  trip 
reports,  FLEX  statement  of  work,  contract  developer’s  system/segment 
specification  (SSS)  and  system  software  design  documents,  written 
change  requests,  and  a  variety  of  Air  Force  manuals  and  support 
materials  on  air  refueling  operations  were  reviewed. 

•  Interviews  -  interviews  were  conducted  with  RL  team,  the  contract 
development  teams,  FLEX  working  group  (operational  personnel  from 
major  commands),  and  tanker  operations  personnel  from  Griffiss  AFB’s 
509th  Air  Refueling  Squadron. 

•  Observation  -  observations  were  made  of  FLEX  Working  Group 
(FWG)  officers  interacting  with  early  prototype  versions  of  the  FLEX 
interface. 

The  user,  organization,  task,  and  environmental/situational  models  evolved  into  a 
set  of  cognitive  task  requirements  (CTRs)  that  became  the  design  objectives  for 
the  interface  prototype.  These  materials  were  used  to  iteratively  refine  the 
models  of  the  air  refueling  domain,  the  TDO  and  the  tanker  re-planning  tasks. 

The  remainder  of  this  section  reports  the  requirements  gathered  from  doc¬ 
umentation,  interviews  and  observation.  A  number  of  graphic  hierarchies  and 
taxonomic  models  were  created  as  the  requirements  evolved.  These  were  used  to 
develop  an  understanding  of  the  procedures  and  information  required  to  accom¬ 
plish  the  re-planning  tasks. 

3.2.1  Defining  the  FLEX  Environmental/Situational  Context 

It  was  possible  to  characterize  the  FLEX  environmental/situational  context  in 
terms  of  its  inherent  structure,  determinacy,  boundedness,  and  complexity.  In 
combat  situations,  decision-makers  in  the  COD  must  cope  with  an  environment 
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that  ranges  from  severely  stochastic  (e.g.,  the  coordination  of  a  complex  array  of 
friendly  assets)  to  indeterminate  (e.g.,  mission  perturbations  caused  by  an 
intelligent  adversary).  There  is  a  high  degree  of  variability  in  ail  the  ATO  plan 
components.  For  example,  the  decision  variables  are  generally  representative, 
but  differ  substantially  in  reliability  due  to  timeliness  of  updates  or  their  inherent 
ambiguities.  Due  to  factors  such  as  uncontrollable  environmental  conditions  and 
the  existence  of  intelligent  adversaries,  it  not  possible  to  completely  control  the 
outcomes  by  manipulating  the  initial  conditions.  Thus,  the  re-planmng 
environment  tends  to  be  open  and  ranges  from  semi-structured  to  unstructured 
due  to  the  high  volume  of  information  and  potential  for  “unknown  unknowns”. 
Most  of  the  information  load  under  routine  conditions  is  tractable  for  a  well- 
trained  and  highly-motivated  TDO.  Under  combat  conditions  (e.g.,  2000  sortie 
ATO)  the  tasks  become  intractable,  with  information  loads  exceeding  human 
ability  to  absorb  and  manipulate. 

Situational  complexity  ranges  from  moderately  high  to  very  high  depending  upon 
the  nature  and  size  of  the  operation.  Air  refueling  is  a  pervasive  support  activity 
and  tanker  missions  are  the  “tent  pole”  in  air  operations.  Moreover,  tanker 
operations  involve  a  secondary  network  of  dependencies.  The  fuel  a  tanker  has 
available  for  refueling  (i.e.,  taskable  fuel)  is  dependent  upon  actual  fuel  offloads 
that,  in  turn,  are  dependent  upon  the  specific  receiving  aircraft  and  the  nature  of 
the  missions  involved.  An  inability  to  meet  refueling  requirements  will  result  in 
cancellation  of  missions  (direct  dependency)  with  a  ripple  effect  upon  the 
missions  which  the  canceled  missions  support  (indirect  dependencies).  Due  to 
these  extended  dependencies,  the  situational  picture  becomes  less  reliable  as 
multiple  changes  to  the  ATO  are  effected  during  combat  execution.  As  a  result, 
the  question  is  not  whether  the  ATO  will  unravel,  it  is  how  much,  in  what  ways, 
and  when  it  will  unravel. 

This  complex  situational  context  has  several  impacts.  First,  in  response  to  the 
domain,  the  organization  must  develop  the  means  to  make  most  efficient  use  of 
resources  in  a  succession  of  varying  short-term  situations.  Moreover,  the 
decision-makers  must  be  able  to  rapidly  and  effectively  exploit  opportunities  and 
retain  maximum  flexibility  and  adaptiveness  in  novel  situations.  There  is  a 
potential  for  misallocation  of  resources  due  to  the  latency  between  recognition  of 
the  situation  and  internal  readjustment.  The  adaptive  strategies  required  (e.g., 
rapid  re-tasking)  may  be  difficult  to  coordinate  and  control  due  to  complex 
missions  interdependencies.  Furthermore,  achieving  the  required  flexibility  may 
negatively  impact  the  ability  to  exercise  control.  Finally,  organizational  learning 
may  be  impaired  by  the  lack  of  repeated  experiences. 

To  meet  the  organizational  response  goals  and  potential  errors,  the  decision¬ 
makers  must  be  provided  with  information  to  help  them  understand  the  structure 
of  the  domain  and  current  problem.  For  example  system-level  (i.e.,  tanker 
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operations)  overview  displays  can  relate  functional  relationships  and  provide 
externalized  mental  models  of  the  operational  domain.  Decision-makers  also 
need  the  ability  to  adaptively  filter  information  at  the  required  abstraction  level, 
while  retaining  rapid  access  to  detailed  information. 


3.2.2  Profiling  the  FLEX  Organizational/Doctrinal  Context 

The  COD  is  part  of  a  hierarchical  organization  which  has  both  a  vertically  and 
horizontally  complex  chain  of  command  with  a  moderately-high  interdependency 
between  functional  units.  The  vertical  complexity  shifts  to  very  high  in  joint  and 
combined  operations  that  require  extensive  coordination.  The  control  structures 
in  adaptive  decision-making  organizations  shift  in  response  to  changes  in  the 
decision  requirements.  Thus,  the  general  tendency  toward  the  more  formal 
organization  evidenced  during  routine  operations  shifts  during  crisis  situations  to 
accommodate  the  requirement  for  a  more  flexible  response. 

During  routine  operations,  the  situational  context  is  determinant  to  moderately 
stochastic.  The  threat  is  low  and  the  environment  is  relatively  static  with  longer 
decision  horizons.  As  a  result,  operations  tend  to  be  tightly  controlled  and 
decision-making  is  more  formal.  Responses  to  re-planning  situations  follow 
more  rigid  procedures  based  on  specific  guidance;  therefore,  the  TDO  is  less 
likely  to  exercise  a  high  degree  of  personal  initiative.  Control  is  communication- 
dependent  and  the  communication  delays  between  levels  of  the  hierarchy  lengthen 
the  time  between  decision  and  action.  Routine  operations  afford  little  opportunity 
to  develop  a  range  of  adaptive  responses  as  the  TDO  never  has  to  push  the  system 
to  the  limit.  As  a  result,  during  non-crisis  operations,  the  TDO  may  be  ill- 
prepared  for  a  sudden  shift  in  the  environment  to  a  combat  state. 

In  contrast,  during  crisis  operations  the  situational  context  is  severely  stochastic 
to  indeterminate.  The  adversarial  threat  of  destruction  and  mission  failure  is 
high  and  decision  time  is  greatly  constrained.  To  facilitate  rapid,  adaptive 
responses,  operational  control  is  loosened  such  that  the  informal  problem-solving 
structures  within  the  COD  may  dominate  the  formal  structures.  As  the  COD 
workload  increases,  the  TDO  will  exercise  more  individual  initiative.  Although 
this  provides  the  TDO  an  opportunity  to  extend  his/her  repertoire  of  response 
options,  the  subsequent  relaxation  of  control  may  result  in  local  satisficing  (that 
is,  solving  the  sub-unit  problem  at  the  cost  of  larger  goals).  Intra-COD 
communication  greatly  increases  and  the  central  role  of  tanker  operations  results 
in  a  barrage  of  task  alerts  to  the  TDO.  Communication  delays  may  impair  infor¬ 
mation  gathering  and  decision  implementation  required  for  more  adaptive 
responses. 
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3.2.3  Profiling  the  Tanker  Duty  Officer  (TDO) 

The  profile  of  the  Tanker  Duty  Officer  (TDO)  incorporates  not  only  their 
knowledge  of  the  specific  functional  tasks  assigned  to  them  and  their  ability  to 
operate  the  system,  but  also  their  understanding  of  goals  and  characteristics  of  the 
larger  domain  in  which  those  tasks  are  performed.  The  TDO  is  typically  an  Air 
Force  major  or  lieutenant  colonel  with  a  moderately  high  knowledge  of  the  air 
operations  domain  acquired  through  experience,  training,  and  service  schools. 

Many  of  the  errors  in  situation  assessment  may  be  traced  to  the  DM’s  knowledge 
of  the  operational  context.  The  TDO  will  have  situational  models  of  the  domain 
mostly  gained  through  instruction  and  exercises  and  should  recognize  most 
prototypical  situations.  In  many  cases,  for  example,  the  TDO  may  have  wing- 
level,  but  not  force-level  mental  models.  TDOs  without  operational  expenence  at 
the  wing  or  force  level  will  not  generally  possess  wholistic  domain  models.  In 
addition,  although  domain-knowledgeable  TDOs  may  exhibit  the  ability  to 
intuitively  interpret  novel  situations,  they  may  not  be  consistent  in  their  com¬ 
bination  of  situational  cues.  TDOs  will  generally  structure  goals  based  upon 
learned  procedures,  direct  guidance,  and  situational  models  of  domain  and  task. 
The  extent  of  his/her  domain  understanding  may  limit  the  TDO’s  ability  to 
resolve  conflicts  between  situational  models.  Situations  triggering  multiple 
models  may  be  interpreted  based  on  the  model  that  is  more  available  or  vivid  in 
memory.  Finally,  the  TDO  may  fail  to  recognize  the  degree  of  uncertainty  in 
current  information  or  the  impacts  of  aggregated  uncertainties  on  the  viability  of 

the  plan. 

The  TDOs’  knowledge  of  the  specific  functional  tasks  assigned  them  in  the  COD 
may  also  vary  depending  upon  their  previous  experiences  in  combat  operations 
(force  and  wing  level)  and  training  (schools  and  exercises).  TDOs  will  typically 
exhibit  high  ability  to  perform  routine  procedures  and  moderate  to  moderately- 
high  adaptability  under  increased  workload  and  novel  situations.  Their  moderate 
to  high  task  experience  potentially  triggers  errors  associated  with  the  heuristics 
used  to  reduce  the  high  workloads  during  ATO  execution  (Table  B-8).  For 
example,  in  high  information  volume  situations,  moderately  knowledgeable  TDOs 
may  not  have  adequate  schema  to  distinguish  relevant  versus  irrelevant 
information.  They  may  also  erroneously  focus  on  task  features  that  match  stored 
(especially  readily  available)  schema.  Fixation  on  task  features  that  match  well- 
known  (or  vividly  remembered)  situations  may  prevent  the  TDO  from  correctly 
diagnosing  the  situation.  Furthermore,  misdiagnosis  may  result  in  the 
misapplication  of  a  learned  response.  More  experienced  TDOs  are  still  vulner¬ 
able  to  a  general  insensitivity  to  the  potential  aggregation  of  error  in  the 
microdecisions  performed  in  multi-stage  decision-making.  For  example,  they 
may  tend  toward  overconfidence  in  their  current  decisions  and  fail  to  revise  their 
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assessments  and  decisions  when  the  situation  changes.  Finally,  there  is  a  general 
tendency  for  the  TDO  to  think  in  serial,  linear  sequences  rather  than  parallel 
networks  of  contributing  causes  and  branching  consequences  of  actions  that  make 
up  the  current  situation  and  affect  the  success  of  the  plan. 

The  TDOs’  system  interaction/operation  knowledge  will  typically  be  the  most 
variable  dimension.  In  the  absence  of  a  protracted  war,  the  majority  of  the 
officers  assigned  to  the  COD  will  be  casual  to  competent  system  users.  That  is, 
they  will  not  routinely  have  to  operate  the  system  under  the  time-critical,  high 
workload  conditions  which  characterize  combat  operations.  Adequate  operation 
of  the  system  during  routine  or  training  operations  will  deteriorate  under  stress 
resulting  in  a  variety  of  errors  and  an  increased  level  of  frustration  and 
confusion.  Casual  system  users  tend  to  forget  training  without  use  and  make 
mistakes  (errors  due  to  wrong  intentions)  and  slips  (errors  due  to  unintentional 
actions).  Casual  system  users  rarely  remember  the  system  shortcuts  that  speed  up 
performance  of  learned  procedures  and  the  increased  workload  will  result  in 
greatly  impaired  performance  for  all  but  simplest  tasks.  The  competent  user  will 
be  able  to  adapt  well-understood  processes  to  increased  workload,  but  still  have 
difficulty  with  the  increase  in  novel  situations.  More  competent  users  make 
mistakes  by  misapplying  learned  procedures. 

TDOs  with  less  system  experience  may  be  confused  by  their  system  operation 
errors.  For  example,  TDOs  may  make  modal  errors  due  to  a  misunderstanding 
about  current  system  state.  A  modal  error  involves  the  incorrect  use  of  an 
interaction  procedure  that  would  be  correct  in  another  system  state.  In  addition, 
users  may  “get  lost”  in  the  system,  finding  themselves  in  unfamiliar  windows  or 
locked  out  while  the  system  performs  an  unintended  procedure. 


3.2.4  Profiling  the  TDO’s  Functional  Tasks 

The  TDO  functional  tasks  were  reviewed,  filtering  them  through  the  user, 
organization,  and  situational  context  profiles  described  above.  This  process 
identified  several  key  dimensions  which  defined  task  performance  and  error 
modes,  including: 

•  Task  complexity  and  difficulty 

•  Task  performance  precision  and  accuracy  requirements 

•  Input  and  feedback  uncertainty 

•  Task  workload  and  potential  stress  dimensions 

It  should  be  noted  that  probing  task  dimensions  often  triggers  further  refinement 
of  the  other  profiles  and  all  of  this  investigation  involved  repeated  iteration  in 


36 


both  top-down  and  bottom-up  analyses.  Figure  14  presents  one  of  the  conceptual 
maps  created  to  describe  the  response  requirements  associated  with  real-time 
threats  that  “pop  up”  during  ATO  execution. 


Figure  14:  Response  to  Real-Time  (“pop-Up”)  Threats 
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3.2.4. 1  Task  Output 

The  TDOs’  discrete  output  unit  is  the  response  to  a  task  request  for  air  refueling 
(AR)  support.  In  a  larger  sense,  the  task  output  is  also  the  overall  status  of  the 
air  refueling  plan  or  the  tanker  operations  system.  The  TDO  is  required  to 
respond  to  a  high  volume  of  AR  task  requests  as  rapidly  as  possible;  thus,  they 
tend  to  be  extremely  intolerant  of  slow  system  response  or  highly  complex 
routines  for  relatively  simple  tasks.  Air  refueling  plans  have  multiple 
components  and  TDOs  need  system  supports  to  prevent  their  losing  track  of  all 
relevant  plan  components.  For  example,  decision-makers  need  the  ability  to 
move  through  various  levels  of  detail  and  system  supports  for  structuring  the 
various  components  to  aid  in  analysis. 


3. 2. 4. 2  Task  Response 


The  TDO’s  response  goals  are  to  meet  the  air  refueling  requirements  of  the  ATO 
and  maintain  a  viable  air  refueling  plan  for  as  long  as  possible.  Both  the  short¬ 
term  execution  goals  and  overall  mission  completion  goals  are  very  difficult  to 
attain.  The  system  should  be  designed  to  offload  the  TDO  of  as  much  of  the 
workload  as  possible  (e.g.,  by  allocation  of  table  look-up  and  computational  tasks 
to  machine).  Some  of  the  subtasks  (e.g.,  keeping  track  of  taskable  fuel)  require 
high  precision  that  is  best  allocated  to  the  machine  component.  For  example,  the 
detailed  data  required  for  response  precision  can  be  maintained  and  manipulated 
by  machine.  In  addition,  automated  updates  relieve  the  TDO  from  being 
overwhelmed  by  the  detail. 

TDO  response  frequency  during  the  execution  of  a  major  combat  ATO  is  veiy 
high.  As  a  result,  AR  tasks  and  changes  to  tanker  operations  pile  up  and  must  be 
prioritized  to  ensure  the  most  important  are  handled  as  rapidly  as  possible. 
Delays  in  feedback  (external  or  internal  to  COD)  may  impair  the  TDO’s  timely 
response. 


3.2. 4.3  Procedures  &  Sub-Tasks 

AR  tasks  arrive  as  discrete  messages,  but  may  have  to  be  handled  by  considering 
the  planning  implications  of  several  changes  simultaneously.  Handling  a  single 
AR  task  involves  several  steps,  including  the  possibility  of  activating  a  ground 
alert  tanker  mission  or  creating  a  new  tanker  mission  to  resolve  major  changes  to 
the  AR  plan.  In  addition,  the  TDO  may  have  the  current  working  task 
interrupted  by  a  higher  priority  task.  The  requirement  for  the  TDO  to  simulta- 
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neously  handle  the  current  AR  tasks  using  FLEX  while  remaining  a  part  of  the 
off-line  COD  activity  (e.g.,  incoming  messages  from  other  sources,  conversations 
with  other  duty  officers,  etc.)  also  contributes  to  the  time  pressure  experienced. 
The  system  must  support  the  TDO’s  maintenance  of  situational  awareness  and  task 
continuity,  and  complement  the  team  activities  of  the  COD. 

AR  subtasks  are  moderately  dependent  in  terms  of  temporal  order  (either  due  to 
system  or  procedural  constraints)  and  logical  relationships;  however,  the  subtasks 
are  highly  dependent  with  respect  to  the  total  AR  plan.  The  overall  dependency 
of  AR  plan  is  such  that  the  complexity  of  relationships  exceeds  the  TDO’s  ability 
to  handle  without  support.  The  TDO  needs  a  way  to  “step  back  from  the  current 
situation  to  see  the  AR  plan  as  a  whole  and  understand  the  various  direct  and 
indirect  dependencies.  AR  tasks’  procedural  complexity  is  moderately  high  to 
very  high  due  to  the  number  of  subtasks  potentially  involved  and  the  depen¬ 
dencies  between  them.  Certain  subtasks  require  strict  adherence  to  set  proce¬ 
dures;  other  subtasks  may  be  handled  in  so  many  ways  that  a  strict  procedure  is 
not  prescribed.  Where  strict  adherence  to  procedures  is  required,  the  system 
support  must  be  designed  to  constrain  TDO  from  ignoring  critical  procedures 
and  make  those  constraints  visible  to  the  TDO.  In  contrast,  where  flexibility  is 
allowed,  the  system  should  facilitate  the  TDO’s  ability  to  manipulate  the  options 
and  make  the  affordances  visible. 


3. 2. 4. 4  Task  Input 

Many  of  the  input  variables  in  the  AR  task  are  moderately  predictable  due  the 
consistency  of  operational  procedures,  basic  situational  stability,  etc.  Some  input 
values  vary  widely  in  predictability  due  to  inaccuracy  of  supporting  data  or 
novelty  of  the  situation.  As  a  result,  the  TDO  may  need  to  be  reminded  of  the 
less  predictable  aspects  of  the  task  to  ensure  that  proper  attention  has  been  paid  to 
the  immediate  contingencies  (“what-ifs”).  For  example,  variations  which  follow 
known  patterns  under  certain  conditions  may  be  stored  as  templates  to  support 
faster  recognition. 

AR  tasks  are  triggered  in  a  very  irregular  fashion;  the  TDO  generally  cannot 
predict  the  flow  of  AR  tasks  with  other  than  very  gross  metrics.  The  TDO 
cannot  control  the  occurrence  of  the  stimulus  (AR  task),  but  can  control  the  order 
of  response  among  tasks  of  the  same  priority.  Although  alarms  may  be  shut  off 
and  incoming  AR  tasks  acknowledged  and  set  aside  for  later  response,  an  AR  task 
remains  an  open  issue  until  changed  by  the  TDO’s  response.  Thus,  the  TDO  may 
need  to  regularly  review  open  requests  and  reorder  priority  under  heavier  work¬ 
loads. 
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3. 2. 4. 5  Task  Feedback 


More  than  50%  of  the  AR  subtasks  involve  decisions  based  on  feedback  from 
previous  responses.  As  suggested  above,  the  TDO  must  respond  to  some  high 
priority  AR  tasks  immediately,  while  other  tasks  may  be  postponed  temporarily. 
For  this  reason,  the  TDO  needs  to  know  when  tasks  will  become  critical  to  help 
in  prioritizing  numerous  tasks  with  the  same  priority.  Feedback  to  the  TDO 
from  other  COD  duty  officers  on  actions  taken  is  immediate;  however,  feedback 
from  the  tankers  and  other  flying  missions  may  be  delayed  by  hours.  As  a  result, 
feedback  reference  may  be  ambiguous  as  actions  taken  early  in  ATO  day  may  be 
superseded  by  later  events  before  feedback  reaches  the  TDO. 

As  the  ATO  day  progresses,  TDO  plan  refinements  may  be  entirely  dependent 
upon  the  projected  effects  of  plan  changes  for  which  there  has  been  only  partial 
feedback.  The  required  reaction  time  for  decisions  is  much  less  than  the  typical 
feedback  lag  and  the  TDO  may  have  to  make  many  dependent  decisions  long 
before  feedback  on  one  decision  is  received.  This  can  result  in  over-  or  under¬ 
adjustments  to  the  AR  plan.  To  compensate,  the  TDO  needs  a  means  to  model 
potential  effects  of  actions  against  a  likely  model  of  the  current  situation.  The 
secondary  effects  of  feedback  lag  impact  the  effectiveness  of  the  decision-maker’s 
learning  and  experience.  False  assumptions  due  to  feedback  lag  can  generate 
inaccurate  mental  models  regarding  cause  and  effect  relationships.  For  this 
reason,  the  TDO  needs  support  for  trying  (and  retracting)  optional  courses  of 
action  before  committing  to  decisions. 


3.2.5  Profiling  the  TDO’s  Decision-Making  Tasks 

The  general  characteristics  of  the  FLEX  functional  tasks  apply  to  all  the  duty 
officer  positions.  For  this  reason,  most  of  the  functional  task  identification 
described  above  was  accomplished  before  the  case  study  was  narrowed  to  tanker 
re-planning  operations.  As  the  requirements  identification  shifted  to  the  detailed 
profiles  of  the  decision-making  tasks,  the  focus  narrowed  to  the  Tanker  Duty 
Officer  (TDO)  with  particular  emphasis  on  the  decision-making  activities 
involved  in  re-planning  during  ATO  execution.  Figure  15  presents  one  of  the 
conceptual  models  developed  to  help  identify  the  key  activities  and  variables  in 
tanker  re-planning  tasks.  This  section  presents  the  decision-making  requirements 
identified  and  modeled.  The  TDO’s  cognitive  task  requirements  are  considered  in 
terms  of: 


•  Stimulus  -  situational  input 

•  Hypothesis  -  situation  interpretation 

•  Option  -  course  of  action  review  and  selection 

•  Response  -  coordination  and  execution  of  chosen  option 
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Figure  15:  Replanning  Tasks 
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3.2.5. 1  Stimulus  -  Characteristics  of  the  Situational  Context  and 
Data  Inputs 

Situation  monitoring  for  the  Tanker  Duty  Officer  (TDO)  in  the  Combat 
Operations  Division  (COD)  is  largely  reactive.  Unlike  real-time  tactical 
monitoring,  the  TDO  is  not  directly  manipulating  the  environment  on  a  minute- 
to-minute  basis.  Instead,  monitoring  and  decision-making  are  carried  out  in  a 
time-constrained  environment,  primarily  driven  by  incoming  update  alerts  or 
task  requests.  Because  important  operations  information  may  exist  on  multiple 
screens,  the  TDO  needs  to  have  changes  brought  to  his  attention.  Pop-up  display 
of  new  task  requests  makes  detection  of  discrete  air  refueling  (AR)  requests 
automatic;  however,  the  TDO  may  have  considerable  difficulty  detecting 
underlying  trends  in  tanker  operations  due  to  variations  in  the  timeliness  of 
updates  to  key  variables. 

Tanker  operations  information  is  primarily  quantitative;  qualitative  information 
is  inferred  through  maps  and  mission  flows.  The  TDO’s  situational  awareness 
requires  supports  for  tailoring  displays  to  filter,  sort,  and  organize  information. 
In  combat  situations,  the  volume  of  incoming  updates  to  tanker  operations  data 
exceeds  the  human’s  ability  to  absorb  or  manipulate  within  the  time  requirements 
The  FLEX  system  automates  the  detailed  updates  and  alerts  the  TDO  to  conflicts 
spawned  by  changes  in  resource  availability. 

FLEX  information  on  tanker  operations  exists  primarily  as  detailed  data  tables 
with  summary  information  available  in  the  Tanker  Status  Display  Board.  The 
Map  Graphic  window  charts  information  such  as  the  locations  of  bases,  tanker 
orbits  and  tracks,  routes  of  planned  missions,  and  defensive  coverage.  FLEX 
users  can  filter  the  information  presented  to  suit  the  requirements  of  their 
decision  tasks.  The  Marquee  is  a  graphic  interface  to  much  of  the  FLEX 
database.  The  Marquee’s  adaptable  display  presents  some  of  the  operational 
dependencies  across  the  ATO  timeline  through  a  database  feature  that  allows  the 
user  to  sort  and  “bundle”  dependent  missions.  However,  the  FLEX  filtering  does 
not  adequately  reduce  workload  due  to  complexity  and  information  volume.  Due 
to  the  screen  layouts  (particularly  in  the  Tanker  Worksheet),  the  TDO  is  still 
required  to  do  some  mental  computation  and  make  notes  to  keep  track  of  certain 
variables.  The  TDO  needs  system  support  to  reduce  off-line  mental  computation 
and  other  memory  requirements. 

Tanker  operations  decision  variables  (e.g.,  fuel  requirements,  etc.)  are  generally 
understood  and  representative.  When  the  required  data  are  current,  the  variables 
are  reliable  for  calculation  and  decision-making;  however,  this  is  not  always 
possible  due  to  communication  failures  or  other  feedback  delays.  Furthermore, 
the  TDO  may  not  fully  assess  the  impacts  of  situation  and  options  based  on 
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displayable  information;  there  are  potential  “unknown  unknowns”  in  combat 
operations  which  undermine  the  representativeness  and  reliability  of  standard 
decision  variables.  Mis-perception  of  the  situation  due  to  incomplete  or 
ambiguous  information  can  lead  to  any  or  all  of  the  following: 

•  Focus  on  irrelevant  information 

•  Selection  and/or  fixation  on  an  incorrect  explanation  or  solution 

•  Incorrect  interpretation  of  cues 

•  Insensitivity  to  missing  information 

Given  these  potential  cognitive  failures,  the  TDO  may  benefit  from  displays  of 
system  models  or  goal  states  to  aid  in: 

•  Identifying  problems 

•  Defining  causal  relationships; 

•  Identifying  missing  information; 

•  Interpreting  ambiguous  cues;  and 

•  Reducing  over-confidence  in  decisions  based  on  uncertain  information. 
The  existing  FLEX  interface  addresses  some,  but  not  all  of  these  needs. 

3.2.S.2  Hypothesis  -  Situation  Assessment  Task  Characteristics 

Several  factors  combine  to  make  hypothesizing  for  situation  assessment  difficult. 
Although  the  TDO  is  familiar  with  all  the  activities  of  tanker  operations,  there  is 
situational  novelty  inherent  in  the  ways  the  variables  may  combine  in  combat. 
Joint  service  and  multi-national  (combined)  operations  add  extra  layers  of 
complexity  and  novelty  to  tanker  operations.  Finally,  the  unpredictability  of  an 
intelligent  adversary  may  result  in  an  unfamiliar  sequence  of  events.  The 
combination  of  novelty  with  the  crush  of  information  flow  may  distract  the  TDO 
from  seeing  the  underlying  similarity  to  more  familiar  situations.  To  relieve  the 
TDO,  certain  routine  aspects  of  AR  re-planning  may  be  allocated  to  machine  pro¬ 
cesses. 

Situation  assessment  for  air  refueling  operations  is  semi-bounded  with  a  moderate 
number  of  hypothetical  possibilities  to  explain  current  AR  plan  status;  however, 
the  number  of  hypotheses  may  seem  greater  under  heavy  workload  situations. 

The  TDO  needs  relief  from  complex  detail  through  aggregated  displays  and 
interaction  with  models  that  help  to  identify  the  differences  between  the  current 
and  goal  states.  Goal-oriented  displays  of  tanker  operations  also  help  to  maintain 
focus  on  critical  variables  and  serve  as  templates  for  analogies  to  familiar 
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situations.  Finally,  to  understand  the  potential  direct  and  indirect  effects  of  the 
current  situation,  the  TDO  needs  a  means  of  viewing  the  consequences  of  actions 
across  the  ATO  day. 

TDO  performs  situation  assessment  tasks  in  a  time-critical,  quasi-real  time 
environment.  This  requires  prioritizing  backlogged  tasks  and  often  means  trad¬ 
ing  off  time  to  fully  analyze  situation  in  order  to  process  more  AR  tasks  in  a 
shorter  period  of  time.  Comments  for  the  FLEX  Working  Group  (FWG)  after 
all  three  prototype  reviews  indicated  that  the  visual  momentum  involved  in  using 
FLEX  was  still  relatively  low  due  to  the  requirement  to  use  operational  data  scat¬ 
tered  across  several  windows  to  accomplish  any  task.  To  relieve  the  time  pres¬ 
sure  in  situation  assessment,  the  TDO  needs  “at-a-glance”  displays  that  do  not 
require  hunting  or  elaborate  manipulation  of  detail  to  get  to  the  relevant  infor¬ 
mation  quickly.  In  addition,  the  TDO  should  not  be  burdened  with  off-line  com¬ 
putation. 

Most  of  the  inferencing  required  for  AR  replanning  is  within  set  bounds, 
involving  well-known  parameters;  however,  the  complexities  of  multiple 
receivers  and  their  dependent  missions  creates  a  hidden  network  of  inferences 
with  varying  degrees  of  certainty.  This  multi-dimensional  network  of  inferences 
is  very  memory-intensive.  To  compensate,  the  TDO  must  use  workload  reducing 
heuristics  that  may  introduce  bias  errors.  The  TDO  needs  displays  which  support 
inferencing  based  on  accepted  operational  procedures.  In  addition,  supports  for 
option  exploration  should  reduce  the  number  of  inferences  and  relieve  the 
workload  on  TDO  by  portraying  the  current  (and  projected)  state  to  compare 
with  immediate  and  longer-term  consequences  across  the  network  of  tanker 
operation  dependencies. 


3.2. 5. 3  Option  -  Course  of  Action  Decision  Tasks 

The  number  of  possible  options  to  a  given  air  refueling  (AR)  situation  are  semi- 
bounded  (as  to  the  limits  of  available  resources,  etc.),  but  sufficient  in  number 
that  the  TDO  faced  with  a  large  number  of  outstanding  AR  tasks  is  often 
overwhelmed  by  the  resulting  plan  complexity.  In  addition,  AR  mission  goals 
may  shift  several  times  in  a  relatively  short  period  of  time,  requiring  a  re- 
evaluation  of  priorities,  updates,  and  recalculation  of  projected  changes  in  AR 
plans.  Most  of  the  conflicts  and  effects  are  predictable,  but  the  number  of 
conflicts  spawned  in  interdependent  missions  by  even  a  small  plan  change  make 
manual  manipulation  intractable.  Furthermore,  the  uncertainties  and  inherent 
complexity  make  outcome  values  for  changing  AR  plans  difficult  to  project 
despite  the  TDOs  understanding  of  the  fundamental  variables. 
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The  TDO  needs  facility  to  quickly  package  responses  for  less  complex,  more 
routine  changes.  The  TDO  needs  some  means  of  rapidly  understanding  the 
fundamental  effects  of  an  option  under  consideration.  Ideally,  the  system  display 
should  support  the  decision-maker’s  rapid  mental  simulation  to  accept  or  reject 
the  option  as  feasible.  Although  evaluating  AR  re-planning  options  is  manually 
intractable  under  high  workload  situations,  the  problem  is  sufficiently  bounded  to 
allow  for  machine  support  in  several  areas,  including: 

•  Rapid  recalculation  of  all  dependent  mission  data  to  compare  options 

•  Mapping  of  restructured  dependencies 

•  Highlighting  any  resulting  conflicts 

To  filter  out  the  best  option  configurations,  the  TDO  needs  tools  that  allow  rapid 
scoring  of  options  against  basic  criteria  with  pre-determined  or  adjustable 
weighting.  Where  rankings  are  similar,  the  TDO  needs  displays  that  model  or 
simulate  the  projected  consequences  for  a  given  option  to  compare  with  other 
relatively  equivalent  options.  Finally,  the  TDO  needs  to  be  able  to  step  back  from 
detail  and  view  AR  operations  in  terms  of  higher  level  goals.  For  example, 
predictable  goal  changes  may  be  combined  into  contingency  scenario  templates 
and  displayed  to  the  TDO  as  advance  notice  or  incoiporated  into  a  rule-based 
advisor. 

Outcome  uncertainty  for  most  AR  plan  components  is  moderate,  but  predictable. 
Nevertheless,  the  broader  the  scope  of  the  plan  change,  the  less  certain  the 
outcome.  TDO  choices  at  time  t  may  leave  them  more  or  less  vulnerable  at  time 
t  +  3.  The  potential  vulnerability  to  later  requirements  changes  (i.e., 
contingencies)  is  even  more  uncertain  and  difficult  to  factor  into  the  decision. 
Combined  levels  of  uncertainty  add  to  the  intractability  of  option  evaluation. 
Moreover,  feedback  may  not  be  timely,  goals  may  change  several  times,  and  there 
is  a  very  high  penalty  for  making  poor  choices.  The  current  FLEX  system  does 
not  reflect  the  uncertainties  aggregated  into  projected  outcomes  of  AR  plans.  The 
system’s  ranking  of  options  treats  all  quantitative  data  as  being  100%  certain. 
Thus,  it  is  possible  to  have  two  equally  ranked  options,  yet  be  unaware  of  their 
highly  disparate  levels  of  certainty.  The  TDO  needs  supports  for  understanding 
the  degree  of  uncertainty  inherent  in  a  particular  option. 


3.2.5.4  Response  -  Planning,  Coordination  and  Execution  of 
Decisions 

Air  refueling  plans  are  operational  hypotheses  involving  multiple  assumptions 
and  inferences  about  the  current  situation  and  the  causal  relationships  that  predict 
outcomes.  AR  execution  in  high  sortie  ATOs  can  make  use  of  pre-planned 
contingencies  (e.g.,  by  activating  orbits  and  routes,  launching  ground  alert  tanker 
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missions,  selecting  alternate  recovery  bases,  etc.)  to  handle  many  of  the  plan 
changes.  Extensive  re-planning  is  required  when  major  changes  are  made  during 
execution  (i.e.,  the  addition  of  a  large,  high-priority  mission;  multiple  failures;  or 
resource  losses).  Re-planning  decisions  are  further  complicated  by  the  difficulty 
of  tracing  all  possible  consequences  of  actions  taken.  The  TDO  needs  support  for 
decomposing  new  goals  into  AR  subtasks  and  means-end  restructuring  of  AR 
plans  to  meet  new  requirements. 

Execution  in  tanker  operations  requires  coordination  with  other  DOs  in  the  COD, 
with  airborne  forward  control  units,  the  affected  strike  wings  and  support 
operations.  During  joint  and  combined  operations  coordination  also  involves 
other  services  and  national  forces.  AR  coordination  must  take  place  within  the 
decision  horizon  and  is  affected  by  the  organizational  shifts  that  occur  in  crisis 
conditions.  Communication  requirements  for  coordination  (i.e.,  management  of 
message  traffic)  impose  processing  loads  on  the  system  which  constrain  the  design 
options.  Reformatting  to  meet  messaging  standards  qualitatively  changes 
information  passed  and  may  affect  its  interpretation  at  the  receiving  end. 

Although  coordination  is  handled  through  SODO  and  ATO  distribution  chain,  the 
TDO  needs  support  for  understanding  the  potential  coordination  ramifications  of 
options  related  to  interdependencies  and  communication  delays. 

Execution  of  AR  plan  changes  is  a  highly  dependent,  multi-phased  control 
process.  Multiple  phases  increase  coordination  requirements  and  can  affect  the 
feasibility  of  certain  options  due  to  the  limits  of  the  decision  horizon.  Delayed 
feedback  may  be  incorrectly  associated  with  the  wrong  phase  and  cause  the  TDO 
to  over-correct.  To  track  execution,  the  TDO  might  benefit  from  a  display  of 
goals  and  subgoals  with  current  execution  status. 

3.2.6  The  FLEX  Cognitive  Task  Requirements  (CTRs) 

Appendix  C  presents  a  summary  of  the  issues  raised  during  the  CTR  identi¬ 
fication  phase  for  the  FLEX  Case  Study.  The  goal  of  the  requirements  identifica¬ 
tion  process  was  to  re-examine  the  available  requirements  definition  resources 
and  enhance  the  existing  FLEX  requirements  specification.  Thus,  many  of  the 
functional  requirements  identified  are  represented  to  some  extent  in  the  FLEX 
System/Segment  Specification  (SSS)  and  the  FLEX  prototypes.  These  high-level 
functional  requirements  for  the  UCI  design  group  under  three  main  support 
requirements:  performance  improvement,  distributed  decision-making,  enhanc¬ 
ing  the  decision-maker’s  knowledge  base.  Table  1  breaks  these  requirements 
down  into  their  respective  components. 
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Support  for  Improved  Performance 

•  Support  rapidly  adaptive  response. 

•  Provide  DM  most  accurate,  relevant  information  and  technological 
means  to  combine  and  interpret  information. 

•  Offload  DM  of  as  much  of  the  workload  as  possible. 

•  Support  pattern-matching,  analogical  reasoning,  and  other  means  for 
improving  assessment  in  novel  situations. 

Support  for  Distributed  Decision-Making 

•  System  must  support  the  TDO’s  maintenance  of  situational  awareness  and 
task  continuity,  and  complement  the  team  activities  of  the  COD. 

•  Provide  means  to  maintain  overall  control  to  meet  mission  objectives 
without  direct  review  of  every  micro-decision  by  senior  command 

•  Optimize  for  fast  communication  to  improve  coordination  and  minimize 
authorization  delays. 

Support  for  Development  of  Decision-Making  Knowledge 

•  Make  use  of  natural  or  domain  knowledge  in  the  interaction  symbology 
to  allow  the  user  to  interact  with  the  task  in  the  most  familiar  terms. 

•  Display  structural  information  (i.e.,  functional  cause  and  effect  relation¬ 
ships)  to  aid  development  of  mental  models  and  support  wider  knowl¬ 
edge  of  response  options. 

•  Provide  doctrinal/procedural  overview  displays  to  support  interpretation 
of  and  effective  response  to  novel  or  rare  events. 

•  Provide  varying  levels  of  explanation  to  support  the  construction  of 
more  robust  mental  models. 


Table  1:  High-Level  Functional  Requirements  for  the  UCI  Design 
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3.2.7  Specific  Cognitive  Task  Requirements 

Appendix  C  presents  a  complete  list  of  the  cognitive  task  requirements  and 
related  issues  raised  during  the  requirements  identification  phase.  It  was  neces¬ 
sary  to  narrow  the  scope  of  the  Tanker  Re-Planning  Case  Study  to  three  key 
CTRs,  unrepresented  in  the  FLEX  SSS  and  unmet  in  the  FLEX  Prototype  3. 
These  included  requirements  to: 

•  Adjust  the  problem  viewpoint  (level  of  detail) 

•  Focus  attention  on  the  key  decision  variables 

•  Compare  response  options  in  terms  of  potential  consequences 

First,  the  TDO  needed  a  way  to  “step  back”  from  the  detailed  data  with  an 
overview  of  tanker  operations.  This  was,  in  part,  a  response  to  the  time  horizon 
of  the  TDO’s  decisions  and  the  varying  degrees  of  timeliness  and  precision  con¬ 
nected  with  the  updates  to  the  database.  Small  changes  to  the  published  ATO 
which  must  occur  rapidly  (e.g.,  last-minute  re-routing  of  a  mission  to  another 
tanker  for  refueling)  are  handled  in  the  air  by  forward  controllers.  The  TDO 
makes  decisions  involving  a  somewhat  longer  decision  horizon  and  needs  to  work 
with  an  aggregated  display  of  the  entire  ATO  day.  Second,  the  TDO  needed  a 
display  simultaneously  presenting  all  the  critical  decision  factors.  The  working 
group  participants  complained  that  key  information  was  distributed  across  several 
displays,  requiring  the  user  to  jump  around  and  make  notes  off-line.  Finally,  the 
TDO  needed  a  support  for  mentally  simulating  the  chain  of  consequences  (e.g., 
changes  in  critical  values)  associated  with  feasible  options.  Answering  these 
requirements  without  sacrificing  access  to  detail  became  the  central  goal  of  the 
interface  re-design. 

The  complete  list  of  cognitive  task  requirements  presented  in  Appendix  C  was 
integrated  into  the  FLEX  System/Segment  Specification  (SSS).  It  was  also  used  to 
distill  the  design  goals  for  the  based  UCI  prototype. 

3.3  Integrating  Cognitive  Task  Requirements  into  the  System 
Requirements  Document 

The  Department  of  Defense  development  standard  for  software  systems  specifies 
the  format  and  content  of  system-level  requirements  documented  in  a  system/ 
segment  specification  (SSS)  document.  Although  the  FLEX  case  study  focused  on 
the  decision  activities  of  the  Tanker  Duty  Officer,  the  CTRs  had  to  be  identified 
and  represented  in  the  higher  level  format  of  the  FLEX  SSS.  This  integration 
involved  distilling  the  findings  from  the  requirements  review  presented  in 
Appendix  C  and  matching  them  to  the  relevant  system  specifications  in  the 
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existing  FLEX  SSS.  In  many  cases,  the  FLEX  SSS  already  contained  statements 
which  incorporated  the  content  of  the  CTR.  Occasionally,  the  statements  were 
modified  to  improve  their  precision.  In  addition,  items  were  appended  to  stated 
requirements  to  detail  functionality  specified  by  identified  CTRs. 

For  example,  feature  visibility  --  facilities  which  enable  the  operator  to  control 
the  visibility  of  all  feature  overlays  (i.e.,  to  enable  or  disable  display  of  feature 
data)  --  include: 


Requirements 

a.  The  operator  shall  be  able  to  select  the  visibility  of .  .  . 

b.  The  operator  shall  be  able  to  create,  store  and  select 
preferred  feature  visibility  defaults  to  filter  or  highlight 
missions/features,  including: 

1.  Specific  ATO  time  range  (current  or  near  future 
operations) 

2.  Missions /features  affected  by  change/update 

3.  Missions/features  in  conflict  (current  or  projected 
conflict) 

Figure  16:  Example  of  a  CTR  Integrated  in  the  FLEX  SSS  Document  (Additional  Tasks 
Requirements  Appear  in  Bold) 


3.4  Translating  Requirements  to  an  UCI  Design  Concept 

The  cognitive  task  analysis  repeatedly  raised  certain  cognitive  aiding  issues. 

These  cognitive  aiding  requirements  aggregate  into  categories  of  design  goals 
representing  situational  awareness  and  understanding,  attentional  focus,  reduction 
of  mental  workload,  problem  perspectives,  option  evaluation,  decision  control 
and  guidance,  interface  operation  and  error  control.The  last  two  goals  involved 
requirements  that  were  adequately  addressed  in  the  existing  FLEX  prototype  and 
lay  outside  the  specific  interests  of  this  research. 


The  remaining  six  belong  to  the  general  category  of  improving  decision-making. 
These  requirements  were  addressed  to  some  degree  in  the  FLEX  SSS  and  the 
FLEX  prototype  designs. 


Each  is  re-capped  briefly  below. 
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Goal  1:  Support  for  Situational  Awareness  and  Understanding 

•  Provide  display  features  (e.g.,  overview  screens)  to  help  the  user 
develop  mental  models  of  the  operational  environment. 

•  Make  the  sources  and  extent  of  uncertainty  explicit. 

•  Provide  templates  of  various  known  patterns  and  causal  conditions  to 
support  faster  recognition. 

Goal  2:  Support  for  Focus  on  Goal/Decision-Relevant  Information 

•  Provide  goal-  or  decision-oriented  displays  to  focus  attention  on  rele¬ 
vant  information  and  support 

••  Identifying  the  situation  and/or  problem; 

••  Defining  causal  relationships; 

••  Identifying  missing  information; 

••  Interpreting  ambiguous  cues;  and 
••  Reducing  over-confidence  in  decisions  based  on  uncertain 
information. 

9  Provide  predictable  goal  changes  in  contingency  scenario  template 
displays. 

Goal  3:  Support  for  Understanding  of  Operational  and  Domain 
Dependencies 

•  Provide  system-level  (i.e.,  tanker  operations)  displays  to  convey  inter¬ 
dependencies  and  situational  overviews. 

•  Example:  the  TDO  needs  ability  to  display  integrated  tanker-receiver 
dependencies,  mission  flows  on  all  active  tanker  orbits  and  fuel  avail¬ 
able. 

Goal  4:  Support  for  Reducing  Mental  Workload 

•  Provide  system  support  to  reduce  off-line  mental  computation  and  other 
memory  requirements. 

•  Provide  an  option  to  use  supports  (e.g.,  table  look-up  tasks)  and 
reminders. 

•  Provide  and  propagate  automated  updates  to  relieve  the  TDO  of  the 
overwhelming  task  of  maintaining  detail. 
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Goal  5:  Support  for  Viewpoint  Adjustment 

•  Provide  the  TDO  the  ability  to  adaptively  filter  information  to  permit 
the  required  abstraction  level,  while  retaining  rapid  access  to  detailed 
information. 

•  Provide  the  ability  to  “step  back”  from  detail  and  view  AR  operations  in 
terms  of  higher  level  goals  and  the  various  direct  and  indirect  depen¬ 
dencies. 

•  Provide  “at-a-glance”  displays  that  do  not  require  hunting  or  elaborate 
manipulation  of  detail  to  get  to  the  relevant  information  quickly. 

Goal  6:  Support  for  Option  Comparisons 

•  Provide  a  means  of  viewing  the  consequences  of  actions  (including  the 
indirect  effects)  across  the  ATO  day. 

•  Provide  support  for  trying  (and  retracting)  solutions  before  committing 
to  decisions. 

•  Provide  a  means  for  a  rapid  mental  simulation  to  accept  or  reject  the 
option  as  feasible. 

•  Provide  displays  which  support  inferencing  based  on  accepted  opera¬ 
tional  procedures. 

•  Provide  support  for  rapid  scoring  of  options  against  basic  criteria  with 
pre-determined  or  adjustable  weighting. 

•  Provide  displays  that  model  or  simulate  the  projected  consequences  for 
a  given  option  to  compare  with  other  relatively  equivalent  options. 

•  Provide  support  for  understanding  the  degree  of  uncertainty  inherent  in 
a  particular  option. 


In  addition  to  the  immediate  benefit  of  improving  performance,  Goals  1  -  3  have 
the  potential  to  enhance  long-term  performance  by  developing  and  reinforcing 
the  mental  models  that  produce  a  more  robust  decision-maker  knowledge  base. 

The  FLEX  Tanker  Case  Study  focused  on  the  immediate  benefits  of  performance 
improvement  derived  from  the  six  design  goals.  Figure  17  maps  the 
interdependencies  associated  with  the  individual  goals.  Research  indicates  that  the 
quality  of  situation  assessment  and  ability  to  preview  the  effects  of  decisions 
improves  decision  performance  (Klein  et  al,  1992;  Klinger  et  al,  1993;  Raphael, 
1991).  In  particular,  improving  the  DM’s  understanding  of  the  causal  dependen¬ 
cies  that  underlie  a  situation  and  the  consequences  of  a  given  course  of  action  can 
help  to  reduce  decision  error  often  associated  with  complex  decisions  (Cohen  et 
al ,  1985;  Reason,  1990;  Senders  and  Moray,  1991).  The  keys  to  situational 
awareness  and  understanding  lie  in  the  DM’s  ability  to: 
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•  Filter  the  relevant  situational  cues  from  the  complex  barrage  of 
data 

•  Combine  the  cues  to  make  inferences  about  the  situation  (Andriole 
and  Adelman,  1989) 


Selecting  the  appropriate  level  of  detail  and  focusing  on  decision-relevant  infor¬ 
mation  assists  the  filtering  process;  while  an  understanding  of  the  operational  and 
domain  dependencies  --  the  causal  networks  —  provides  a  framework  for 
combining  information  to  make  inferences.  Relieving  the  DM  of  certain  detailed 
mental  operations  (e.g.,  calculations,  table  look-up  operations,  and  various  mem¬ 
ory  tasks)  and  providing  mental  organizers  (e.g.,  decision-structured  displays) 
permits  the  focus  of  mental  resources  on  the  critical  decision  tasks.  Finally,  the 
ability  to  compare  options  in  terms  of  potential  consequences  of  actions  taken  is 
enhanced  by  the  DM’s  focus  and  understanding. 


The  tasks  identified  for  the  FLEX  Tanker  module  during  the  requirements 
identification  phase  and  incorporated  into  the  six  design  goals  above  map  to  four 
CSE  design  principles.  These  principles,  with  the  associated  design  goals  in 
parenthesis,  include: 

Presenting  a  system-level  model  relating  the  relevant  decision  variables 
to  focus  the  decision-maker’s  attention  and  guide  the  selection  of 
appropriate  detail  to  further  inform  the  decision  process  (Goals  1-6); 
Integrating  all  the  key  decision  factors  in  one  display  to  eliminate 
unnecessary  jumping  from  screen  to  screen  (Goals  2  -  5); 

Making  the  current  system  (i.e.,  tanker  operations)  state  visible  to 
highlight  the  areas  requiring  correction  (Goals  2  -  5); 

•  Relieving  the  DM  of  calculation  and  memory  tasks  (Goals  2,  4  and  6); 

•  Making  the  consequences  of  options  visible  for  comparison  and 
evaluation  (Goals  1,  2  and  6). 

The  first  two  principles  were  drawn  primarily  from  the  ecological  interface 
design  research  by  Jens  Rasmussen  and  his  colleagues  (Rasmussen  and  Vicente, 
1989;  Vicente  and  Rasmussen,  1992)  and  represented  in  guideline  form  in 
Rasmussen  and  Pejtersen  (1993)  and  Rasmussen  et  al,  (in  press).  In  addition, 
research  on  the  design  of  integrative  displays  (Bennett  et  al,  1993)  provided  fur¬ 
ther  insight  into  the  ways  decision  cues  can  be  combined  in  symbolic  displays 
whose  decision-aiding  “emergent”  features  are  only  apparent  in  that  combined 
form.  Finally,  the  tactical  decision-making  research  by  MacMillan  and  Entin 
(1991)  illustrated  the  decision  performance  value  of  unifying  the  key  decision 
factors  in  a  single  window.  The  three  remaining  principles  reflect  guidance  that 
may  be  found  in  all  standard  guideline  sources. 
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Figure  17:  Relationship  of  FLEX  Design  Goals  to  Overall  Goal  of  Improved  Decision 
Performance 
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The  guidance  from  these  principles  drove  the  design  of  an  additional  window  for 
the  FLEX  Tanker  DO  called  Option  View.  (Figure  18).  The  Option  View 
window  incorporates  a  number  of  UCI  responses  to  the  design  principles 
identified.  First,  the  window  presents  a  high-level  system  model  of  current 
tanker  operations  displaying  the  active  tanker  missions  at  their  orbit  locations 
across  the  24  hours  of  the  ATO.  The  receiver  contacts  are  mapped  across  time 
against  the  assigned  tanker  mission  to  highlight  their  flow  in  terms  of  density  and 
timing.  Conflicts  are  highlighted  in  red  to  draw  attention;  changes  in  the  tanker 
or  receiver  missions  are  highlighted  in  yellow.  The  taskable  fuel  remaining  is 
displayed  above  each  tanker  mission  and  relieves  the  DM  from  having  to  make 
the  calculation.  Second,  to  facilitate  comparison,  two  options  may  be  compared 
simultaneously  against  the  planned  ATO.  (The  actual  large-screen  monitor  used 
for  the  Air  Force  FLEX  prototype  would  support  comparison  of  more  than  two 
options.)  The  comparisons  present  the  effects  of  allocations  in  terms  of  changes 
to  the  taskable  fuel  remaining,  timing  of  receiver  contacts,  and  density  of 
assigned  receivers  against  the  tanker. 

3.5  Developing  an  Interactive  Prototype  of  the  UCI  Design  Concept 

The  FLEX  ATTD  is  a  technology  demonstration  program  that  is  intended  to 
evolve  into  a  fielded  system.  Given  the  author’s  external  role  in  the  FLEX 
ATTD,  the  FLEX  Tanker  Case  Study  made  use  of  a  throwaway  prototype  to 
evaluate  the  UCI  design  impacts  on  decision  performance.  For  evaluation  and 
comparison,  both  the  FLEX  tanker  module  displays  and  the  revised  UCI  design 
were  implemented  in  an  interactive  prototype.  The  essential  features  of  the 
existing  FLEX  windows  were  mocked-up  to  allow  for  rapid  prototyping  of  the 
key  decision  factors  presented  in  each  window  (Appendix  G).  The  extensive 
searching,  sorting  and  tailoring  capabilities  of  these  displays  were  not  represented 
in  order  to  focus  the  evaluation  on  the  decision-making  tasks  rather  than  the 
interface  manipulation  tasks.  The  evaluation  prototype  was  developed  in 

SuperCard®  on  an  Apple  Macintosh®  with  a  high-resolution  RGB  color  monitor. 
To  facilitate  non-intrusive,  automated  data  collection,  the  software  program 
includes  routines  to  record  time-stamped  information  about  the  user’s  interaction 
with  the  interface. 

3.6  Evaluating  the  UCI  Design  Concept 


In  rapid  prototyping  development  efforts,  software  evaluation  goes  on 
continuously  as  functional  modules  are  developed  and  integrated.  In  similar 
fashion,  UCI  concepts  and  features  may  be  evaluated  early  in  development  as 
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design  hypotheses.  Such  early  evaluation  is  particularly  important  when  the 


Figure  18:  Option  View 


system  contemplated  will  comprise  a  major  change  to  the  decision-making 
organization.  Early  concept  evaluations  are  also  useful  for  evaluating  the  value- 
added  by  incorporating  advanced  UCI  technologies. 

In  addition,  to  the  narrowly  focused  evaluations  conducted  throughout  the  life- 
cycle,  the  overall  UCI  design  must  be  evaluated  as  part  of  a  total  prototype 
evaluation.  This  allows  the  designers  to  examine  the  flow  of  interaction  between 
the  user  and  the  computer  and  explore  interface  problems  that  may  not  surface  in 
limited  studies.  Overall  evaluation  is  best  conducted  using  subjects  that  represent 
a  cross-section  of  the  target  end-user  population.  Although  the  FLEX  Case  Study 
only  focused  on  a  small  subset  of  the  larger  FLEX  system,  the  case  study 
evaluation  was  conceived  in  terms  of  a  complete  review  of  the  UCI  concept  in  the 
prototype. 


3.6.1  Developing  Evaluation  Goals 

The  fundamental  hypothesis  of  the  cognitive  systems  engineering  framework  is 
that  using  the  approach  should  highlight  the  critical  cognitive  task  requirements 
and,  by  guiding  the  translation  of  these  requirements  into  design  concepts,  result 
in  changes  in  the  system  which,  in  turn,  result  in  changes  in  task  performance. 
The  evaluation  of  the  FLEX  Tanker  Module  Prototype  sought  to  validate  the 
approach  by  demonstrating  an  improvement  in  decision  performance  along  three 
dimensions:  situational  awareness  and  understanding,  option  evaluation,  and 
cognitive  workload. 


3.6.2  Selecting  Evaluation  Methods 

The  evaluation  goals  identified  were  very  specific  to  the  cognitive  task 
requirements  and  unique  features  of  the  tanker  operations  domain.  For  this 
reason,  it  was  critical  to  evaluate  the  task  interaction  concepts  as  well  as  the 
information  presentation  aspects  of  the  UCI  design.  The  RL  version  of  the  FLEX 
prototype  did  not  have  facilities  for  setting  up  multiple  small  trials.  More 
importantly,  the  interface  was  both  “fragile”  (i.e.,  prone  to  frequent  crashes)  and 
very  difficult  to  learn.  The  prototype  developed  for  evaluation  focused  on  the 
decision  tasks  and  minimized  system  operation  tasks  by  pre-formatting  the  highly 
customizable  FLEX  windows  so  that  any  window  called  by  the  user  would  display 
its  information  to  best  advantage.  This  was  done  to  eliminate  performance 
variation  due  to  differences  in  system  operation  skills.  The  high  level  of  domain 
and  task  knowledge  that  characterized  the  target  users  suggested  that  subjects  for 
the  interaction  should  be  drawn  from  a  Air  Force  officers  with  a  common  level 
of  knowledge  and  experience  in  tanker  operations. 
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As  indicated  previously,  the  framework  for  the  evaluation  of  the  FLEX  UCI 
design  was  built  upon  a  multi-dimensional  view  of  the  factors  contributing  to 
effective  decision-making  performance.  The  fundamental  hypothesis  for  evalua¬ 
tion  may  be  stated  as  follows: 

UCI  designs  based  upon  the  approach  to  identification  and 
specification  of  cognitive  task  requirements  will  result  in 
improved  decision-making  performance  ... 

This  high-level  hypothesis  was  broken  down  into  measurable  factors  with  respect 
to  three  dimensions:  situational  awareness  and  understanding,  option  evaluation 
and  selection,  and  cognitive  workload.  Each  dimension  was  represented  by  one 
or  more  design  goals  that,  in  turn,  were  the  subject  of  one  or  more  sub¬ 
hypotheses  and  measures.  Figure  19  maps  the  six  evaluation  hypotheses  and 
related  measures  to  these  three  dimensions.  Each  dimension  is  discussed  in  turn 
below. 


Dimension  1:  Situational  Awareness  and  Understanding 

•  Design  Goal:  The  presentation  of  information  was  designed  to  highlight  and 
relate  key  decision  factors  at  the  appropriate  level  of  abstraction  to  relieve 
DMs  from  the  requirement  to  accomplish  this  integration  in  their  heads. 

Hypothesis  1.1a:  Decision-makers  presented  an  integrated  model  of  the 
“system”  and  critical  decision  variables  will  more  accurately  focus  their 
information  search  than  those  not  supplied  with  the  integrated  model  display. 

Hypothesis  1.1b:  In  the  absence  of  a  fully  integrated  model  display,  decision¬ 
makers  will  compensate  by  selecting  the  displays  which  partially  integrate  key 
variables. 

Measures: 

1.  Time-stamped  Process  Trace  of  Information  Views  Used 
(Comparison  with  decision  model  of  where  critical  decision 
information  is  located) 
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Figure  19:  Relationship  of  FLEX  Evaluation  Hypotheses  &  Measures  to  the  UCI  Design  Goals 
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••  Comparison  of  mean  frequency  of  window  selection 
••  Process  trace  (precisely  where  user  went  when) 

••  Comparison  of  mean  duration  (seconds)  spent  viewing  each  window 

2.  Subjective  Interface  Evaluations 

(Comparison  of  interface/task  means  based  upon  users  rating  on  discrete  scale 
of  specific  window’s  usefulness  in  four  decision  tasks) 

••  Problem  Identification  ••  Option  Evaluation 

••  Situation  Assessment  ••  Option  Selection 


Dimension  2:  Option  Evaluation 

•  Design  Goal:  The  information  presentation  and  interaction  was  designed  to 
allow  exploration  and  comparison  of  two  or  more  options  in  terms  of  their 
consequences  across  time. 

Hypothesis  2.0:  Displaying  the  changes  in  the  critical  variables  to  allow 
simultaneous  exploration  of  two  or  more  options  will  improve  option 
evaluation  and  selection  performance. 

Measures: 

1.  Speed  (comparison  of  mean  times  to  make  individual  decision  -  trial  and 
sum  -  by  interface) 

2.  Accuracy 

••  Comparison  of  mean  score  on  selection  of  “better”  option  across 
trials,  users,  and  interfaces 

••  Comparison  of  ANOVA  on  scores  across  trials,  users,  and  interfaces 
(“better”  option  determined  by  previously  established  experts’  model 
rating  options  based  on  taskable  fuel  remaining  and  receiver  “density” 
function) 


NOTE:  Interface  exposure  order  effects  were  compared  to  evaluate  the 
potential  task  and  interface  learning  interaction  across  sessions. 
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Dimension  3:  Cognitive  Workload 


•  Design  Goal:  Reduce  the  users’  experience  of  cognitive  workload  due  to 
mental  demand  and  time-pressure  by  designing  the  information  presentation  as 
a  “system  model”  representing  and  relating  critical  decision  variables. 

Hypothesis  3.1a:  When  other  task  factors  are  held  constant,  the  perceived 
workload  associated  with  time-pressure  and  problem  complexity  will  be 
greater  for  decision-makers  working  without  integrated  displays. 

Measure:  NASA-TLX  workload  assessment.^ 

••  Comparison  of  the  percentage  of  total  workload  attributed  to  temporal  and 
mental  demand  depending  upon  interface  used 

Hypothesis  3.1b:  The  subjective  evaluation  of  interfaces  will  favor  those 
interfaces  associated  with  lower  cognitive  workload  ratings  (i.e.,  those  that 
reduce  task  complexity  in  terms  of  mental  and  temporal  demand). 

Measures: 

1.  NASA-TLX  workload  assessment 

••  Mean  percentages  by  interface 
••  Mean  total  workload  by  interface 

2.  Subjective  Interface  Evaluations 

••  Comparison  of  mean  subjective  evaluations  interface  effectiveness 
across  decision  tasks  (problem  identification,  situation  assessment, 
option  evaluation,  option  selection) 

••  Review  of  open-ended  written  and  verbal  impressions  of  interfaces 
(audio  recording  of  discussion  after  final  session)  vis-a-vis  task 
requirements 

••  Design  Goal:  Display  the  changes  in  the  critical  variables  to  relieve  the 
decision-maker  of  the  extra  cognitive  workload  involved  in  mentally 
simulating  the  comparative  effects  of  the  options.  Allocate  tasks,  such 
as  calculation  of  numerical  values  (e.g.,  fuel  remaining),  to  the  computer 

3  NOTE:  NASA  TLX  is  a  subjective  rating  of  the  user's  perception  of  the  source  of  task  workload 
across  multiple  dimensions  (e.g.,  mental  demand,  temporal  demand,  own  performance,  frustration 
effort,  etc.) 
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as  calculation  of  numerical  values  (e.g.,  fuel  remaining),  to  the  computer 
to  relieve  users  of  mental  calculation. 

Hypothesis  3.2:  Decision-makers  provided  integrated  displays  (i.e.,  those 
presenting  calculations  of  all  key  variables)  for  comparing  the  options  will 
not  make  off-line  notes  to  support  their  mental  simulations. 

Measure:  Direct  observation  -  collection  of  session  materials  for  review 
(i.e.,  did  the  users  make  notes  and  calculate  values  while  using  the  interface) 


The  prototype  evaluation  was  specifically  designed  to  explore  the  constructs 
behind  the  cognitive  task  requirements  and  demonstrate  the  range  of  information 
that  could  be  gathered  and  analyzed  quickly.  The  various  measures  selected  were 
chosen  for  their  presumed  validity  as  measures  of  the  criteria  of  interest,  but 
preference  was  given  to  methods  that  were  either  very  quick  to  analyze  or  could 
be  automated  in  the  software  of  the  interface  prototype.  For  example,  process 
measures  were  chosen  which  could  be  captured  and  compiled  automatically  rather 
than  employing  a  team  of  observers,  transcribers,  and  coders  to  collect  and 
format  verbal  protocols.  The  subjects  were  provided  several  opportunities  to 
comment  on  the  nature  of  the  interaction  and  the  information  presentation.  As 
much  as  possible,  these  subjective  data  were  collected  in  structured  formats  that 
facilitated  rapid  coding  and  analysis. 

Since  a  sufficiently  large  group  of  representative  users  is  difficult  to  obtain  for 
long  periods  of  time,  the  evaluation  sessions  were  designed  to  require  each 
participant  to  commit  to  only  two  half-day  interaction  sessions.  Counter-balanced 
exposure  and  a  repeated  measures  design  provided  sufficient  power  to  achieve 
significant  results  with  a  total  of  twelve  subjects. 

The  results  of  the  evaluation  generally  supported  all  hypotheses.  Comments  from 
the  subjects  after  exposure  to  both  interface  designs  strongly  favored  the  addition 
of  the  Option  View  window.  Moreover,  the  subjects’  difficulties  with  the  tasks 
when  using  the  original  FLEX  interface  conformed  to  the  errors  predicted 
during  the  requirements  identification. 


3.7  The  Cognitive  Task  Analysis  Handbook 

The  concepts,  methods  and  techniques  that  guided  the  design  and  evaluation  of  the 
FLEX  interface  and  prototype  lead  to  the  development  of  a  Handbook  for 
repeatable  UCI  design,  prototyping  and  evaluation.  This  Handbook  appears  in 
Appendix  C. 
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4.0  The  UCI  Design,  Prototyping  &  Evaluation  Workbench 


4.1  Hardware  /Software  Configuration 

DesignPro  is  an  Apple  Macintosh-based  application  that  was  developed  in 
Supercard  (by  Allegiant  Software). 

The  system  also  uses  a  FileMaker  Pro  data  base  of  “snippets”  --  video  clips  of 
UCI  features  stored  as  QuickTime  videos. 

DesignPro  has  embedded  COTS  software  that  supports  the  prototyping  process. 

The  system  was  configured  for  a  1 6”  monitor  and  requires  24K  of  RAM  and 
350K  of  hard  disk  space  to  run  efficiently. 


4.2  Operation 

The  system  is  “user-friendly.”  Those  with  UCI  design,  prototyping  and 
evaluation  experience  will  find  the  interface  intuitive;  those  with  relatively  little 
experience  will  find  the  system  easy  to  use  --  especially  via  its  embedded  tutorial 
(see  below). 

The  following  18  figures  illustrate  DesignPro ’s  capabilities. 

Figure  20  presents  the  master  menu.  It  identifies  three  primary  activities  areas  - 
requirements  modeling,  prototyping  and  evaluation  --  and  a  secondary  area  -- 
UCI  sampling,  which  houses  the  video  snippets  of  selected  UCI  features  and 
COTS  prototyping  software. 

Each  icon  represents  a  functional  area  available  to  the  designer.  The  icons 
represent  a  top-to-bottom  sequential  process,  though  the  designer  is  not  bound  to 
proceed  sequentially. 

The  “UCI  Sampling”  activity  area  can  be  accessed  at  any  time. 

Figure  21  presents  the  domain  menu.  The  only  domain  in  the  system  is 
“command  and  control.”  Other  domains  can  be  added  to  the  system  as  knowledge 
is  modeled  in  the  other  areas. 

The  icons  at  the  bottom  of  the  screen  in  Figure  21  indicate  the  navigational 
capabilities  of  DesignPro. 
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Figure  20:  The  Master  Menu  Structure 
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RURILRBLE  DOMRIN  GUIDANCE 
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Figure  21:  The  Domain  Selector 


Figure  22  takes  the  user  to  the  hierarchical  task  analysis  capability  of  DesignPro. 
This  capability  permits  designers  to  identify  the  tasks  that  the  user  of  the  interface 
will  have  to  perform. 

There  are  several  levels  available  to  the  designer  as  well  as  a  horizontal  capability 
that  permits  5  top  level  tasks,  20  second  level  tasks,  and  60  lower  level  tasks. 

The  “task  map”  at  the  top  of  the  screen  permits  designers  to  travel  across  the  task 
hierarchy. 

The  hierarchical  task  analysis  capability  also  permits  each  task  to  be  annotated. 

Figure  23  indicates  how  the  system  asks  designers  to  characterize  each  task.  A  „ 
series  of  judgments  must  be  rendered  to  permit  the  tasks  to  be  profiled.  High, 
“medium,”  and  “low”  ratings  are  permitted. 

The  system  also  permits  each  judgment  to  be  annotated. 

Figure  24  requires  the  designer  to  identify  the  constraints  that  will  bind  the 
design,  prototyping  and  evaluation  process. 

These  constraints  address  the  following: 

•  Target  Platform 

•  Display  Device 

•  Display  Device  Size 

•  Operating  System 

•  Management 

•  Hard  Disk  Storage 

•  Graphical  User  Interface 

•  RAM 

Data  about  these  constraints  help  determine  what  is  possible. 

Figure  25  provides  the  user  with  some  “cases”  with  which  to  compare  the  design 
problem  at  hand. 

“Case-based  reasoning”  is  a  popular  analytical  technique  that  permits  problem- 
solvers  to  better  understand  a  problem  at  hand  via  the  inspection  of  past  problem¬ 
solving. 

Several  cases  are  embedded  into  the  system;  the  system  can  receive  any  number 
of  cases. 
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HIERARCHICAL  TASK  RNALVSIS 


Figure  22:  Hierarchical  Task  Analysis 


HIERARCHICAL  TRSK  RNRLVSIS 
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Figure  23:  Tksk  Profiling 


CONSTRAINTS 


Figure  24:  Assessment  of  Constraints 


Ruailable  Cases 
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Information  I  Main  Menu 


Figure  26  provides  the  designer  with  a  status  report  on  how  much  of  the  design 
process  has  been  completed  and  what  remains  to  be  done.  If  key  steps  have  not 
been  taken  then  the  system  will  not  permit  the  design  process  to  move  to 
prototyping. 

Figure  27  shifts  to  the  display  design  recommendations  that  the  system  makes 
after  requirements,  constraints  and  user  profile  information  has  been  entered. 
These  recommendations  are  based  on  knowledge  of  the  UCI  design  process.  It 
also  indicates  how  “confident”  the  system  is  in  the  recommendation  with  a  sliding 
scale  that  portrays  the  recommendation  as  “weak”  to  “strong.” 

The  system  also  permits  designers  to  see  examples  of  the  recommended  UCI 
features  as  well  as  linkages  back  to  the  requirements  models. 

Figure  28  repeats  the  process  for  data  and  information  coding  recommendations, 
while  Figure  29  presents  the  dialog  and  interaction  recommendations. 

Figure  30  provides  prototyping  options  (by  platform).  This  is  the  path  to  the 
COTS  tools  that  can  be  used  to  build  interactive  prototypes  that  feature  the  UCI 
display  and  interaction  recommendations  made  by  the  system. 

Figure  31  presents  cases  to  the  designer,  cases  that  can  help  with  the  development 
of  the  prototype. 

Figure  32  again  checks  the  status  of  the  design  and  development  process. 

Figure  33  permits  the  designer  to  recall  previous  inputs  and  inspect  how  they 
“trace”  to  features  and  their  representation  in  the  prototype. 

Figure  34  walks  the  designer  through  the  evaluation  process. 

Figure  35  presents  some  testing  options  to  the  designer. 

Figure  36  presents  another  checklist,  while  Figure  37  provides  another  status 
check. 


4.3  The  FLEX  Tutorial 

Embedded  in  the  system  is  a  tutorial  based  on  the  FLEX  case  study.  This  tutorial 
is  intended  to  introduce  designers  to  the  DesignPro  system  and  to  the  domain  of 
command  and  control,  the  only  knowledge  domain  currently  in  the  system. 
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OUERRLL  REQUIREMENTS  STATUS 
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Figure  26:  Status  Display 


Figure  27:  Display  Recommendations 


Data  &  Information  Coding  Features  Inclusion  Confidence  Leuel  *  Example 
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Figure  28:  Information  &  Data  Coding  Recommendations 


DIALOG  &  INTERACTION 


Figure  29:  Dialog  &  Interaction  Recommendations 


PROTOTYPE  IMPLEMENTATION 
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Figure  30:  Prototyping  Options 


Figure  31:  Cases 


OUERRLL  PROTOTVPING  STATUS 
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Figure  32:  Status  Check 


REQUIREMENTS  TRRCERBILITV 
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Figure  33:  Traceability 


PERFORMANCE  MEASURES  OPTIONS 
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Figure  34:  Evaluation  Process 


Figure  35:  Testing  Options 
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Figure  36:  Checklist 


Figure  37:  Status  Check 


5.0  Summary  &  Conclusions 


This  final  technical  report  covers  the  period  from  August  1,  1992  through 
August  31,  1995.  The  report  describes  progress  made  in  the  development  of  the 
DesignPro  interactive  computer-based  advisory  system  for  user-computer 
interface  (UCI)  design,  prototyping  and  evaluation. 

The  overall  process  includes  interaction  among  knowledge  templates  to  develop  a 
requirements  model  that,  in  turn,  helps  yield  displays  and  UCI  routines  which,  in 
turn,  suggest  a  prototyping  strategy  which,  in  turn,  identifies  evaluation  tactics. 

The  DesignPro  system  supports  to  the  UCI  designer;  it  does  not  call  for  the 
replacement  of  human  UCI  expertise  in  the  design  process.  The  methodology 
assumes  that  commercial-off-the-shelf  (COTS)  software  can  be  used  to  create 
(simulate)  an  integrated  environment  for  designing,  prototyping  and  evaluating 
interactive  user  computer  interaction  routines. 

The  project  was  anchored  in  the  systems  engineering  approach  to  interactive 
systems  design  and  development;  the  throwaway  >  evolutionary  prototyping 
approach  to  validate  requirements  was  implemented.  An  initial  prototype  was 
released  in  January  of  1993;  another  in  April  of  1993  and  another  in  October 
1993;  refinements  were  made  to  the  prototype  in  January  and  March  of  1994  and 
then  again  in  April  1995,  with  a  final  prototype  release  in  August  1995.  The 
prototypes  were  used  to  validate  workstation  design  requirements  and  to 
communicate  what  the  system  does.  They  also  permitted  us  to  integrate  many 
concepts,  tools,  and  COTS  software  programs  into  the  design. 

The  project  has  also  pursued  a  case  study  within  its  scope.  The  FLEX  case  study 
was  completed  during  this  reporting  period  and  presented  to  Rome  Laboratory 
personnel  in  July  1994.  FLEX  illustrated  how  the  UCI  design,  prototyping,  and 
evaluation  methodology  embedded  in  DesignPro  can  be  used  to  design,  prototype 
and  evaluate  varieties  of  command  and  control  interfaces. 

The  DesignPro  Advisory  System  permits  designers  of  user  computer  interfaces  to 
represent  requirements,  to  build  prototypes,  and  to  evaluate  their  impact  —  all  via 
a  “workbench”  of  user  accessible  functions. 

The  following  figure  presents  the  DesignPro  workbench.  Note  the  major 
functional  areas  and  the  system’s  ability  to  show  examples  of  the  features  that 
comprise  user  computer  interfaces  as  well  as  examples  of  off-the-shelf 
prototyping  tools  via  a  “browser”  capability. 
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The  project  synthesized  findings  from  a  variety  of  sources  and  disciplines  -  as 
suggested  by  the  following  figure: 


Human  Factors 
Engineering 
&  Ergonomics 


Domain  Analysis 


Systems 
Engineering, 
Software  Systems 
Engineering  & 
Cognitive  Systems 
Engineering 


User-Computer 
Interface  & 
Interaction  Routine 
Design,  Prototyping 
&  Evaluation 
Workbench 


Information 

Technologies 


Figure  39:  The  Project’s  Analytical  Backdrop 


The  project’s  ultimate  payoff  will  depend  upon  the  nature  of  the  user-computer 
interface  design  applications  to  which  the  workbench  is  applied;  a  large  number 
of  applications  will  provide  insight  into  the  operational  capabilities  of  the 
interface  and  the  analytical  and  design  assumptions  upon  which  it  is  based. 

Ultimately  the  workbench  demonstrates  a  growing  trend  in  the  design  arena:  the 
embedding  of  more  and  more  design  expertise  in  the  knowledge-based  software 
systems  capable  of  -  in  most  cases  --  advising  designers  and  -  in  a  few  cases  - 
automating  design  processes. 
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Appendix  A 

Screen  Displays  from  the  DesignPro  Workbench 
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File  Edit  Go  Tools  Objects  Te«t  «  File 
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Appendix  B 

FLEX  “Before”  &  “After”  Screens 
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Task  Inspector 
Window  (Modified  for  Case  Study) 


Tanker  Worksheet 
FLEX  Window  (Modified  for  Case  Study) 
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Window  (Modified  for  Case  Study) 


Map  Graphic 

FLEX  Window  (Modified  for  Case  Study) 
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Marquee 

Window  (Modified  for  Case  Study) 
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LINKAGE  00  AT  SHELL  FCR  10  FRE  MISSION  FUELING 
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Subject  Tracker 

Automated  Data  Collection  #3:  Process  Tracing 


MISSION 

OF 

AFRL/INFORMATION DIRECTORATE  (IF) 


The  advancement  and  application  of  information  systems  science  and 
technology  for  aerospace  command  and  control  and  its  transition  to  air, 
space,  and  ground  systems  to  meet  customer  needs  in  the  areas  of  Global 
Awareness,  Dynamic  Planning  and  Execution,  and  Global  Information 
Exchange  is  the  focus  of  this  AFRL  organization.  The  directorate’s  areas 
of  investigation  include  a  broad  spectrum  of  information  and  fusion, 
communication,  collaborative  environment  and  modeling  and  simulation, 
defensive  information  warfare,  and  intelligent  information  systems 
technologies. 


